diff options
| author | Alex Groleau <agroleau@gitlab.com> | 2019-08-27 12:41:39 -0400 |
|---|---|---|
| committer | Alex Groleau <agroleau@gitlab.com> | 2019-08-27 12:41:39 -0400 |
| commit | aa01f092829facd1044ad02f334422b7dbdc8b0e (patch) | |
| tree | a754bf2497820432df7da0f2108bb7527a8dd7b8 /doc | |
| parent | a1d9c9994a9a4d79b824c3fd9322688303ac8b03 (diff) | |
| parent | 6b10779053ff4233c7a64c5ab57754fce63f6710 (diff) | |
| download | gitlab-ce-aa01f092829facd1044ad02f334422b7dbdc8b0e.tar.gz | |
Merge branch 'master' of gitlab_gitlab:gitlab-org/gitlab-cerunner-metrics-extractor
Diffstat (limited to 'doc')
879 files changed, 21257 insertions, 12332 deletions
diff --git a/doc/README.md b/doc/README.md index 25db0efb960..f12c06199c2 100644 --- a/doc/README.md +++ b/doc/README.md @@ -9,7 +9,7 @@ description: 'Learn how to use and administer GitLab, the most scalable Git-base </div> <!-- the div above will not display on the docs site but will display on /help --> -# GitLab Documentation +# GitLab Docs Welcome to [GitLab](https://about.gitlab.com/) Documentation. @@ -25,7 +25,7 @@ No matter how you use GitLab, we have documentation for you. | [**User Documentation**](user/index.md)<br/>Discover features and concepts for GitLab users. | [**Administrator documentation**](administration/index.md)<br/>Everything GitLab self-managed administrators need to know. | | [**Contributing to GitLab**](#contributing-to-gitlab)<br/>At GitLab, everyone can contribute! | [**New to Git and GitLab?**](#new-to-git-and-gitlab)<br/>We have resources to get you started. | | [**Building an integration with GitLab?**](#building-an-integration-with-gitlab)<br/>Consult our automation and integration documentation. | [**Coming to GitLab from another platform?**](#coming-to-gitlab-from-another-platform)<br/>Consult our handy guides. | -| [**Install GitLab**](https://about.gitlab.com/install/)<br/>Installation options for different platforms. | [**Subscribe to GitLab**](#subscribe-to-gitlab)<br/>Get access to more features. | +| [**Install GitLab**](https://about.gitlab.com/install/)<br/>Installation options for different platforms. | [**Customers**](subscriptions/index.md)<br/>Information for new and existing customers. | | [**Update GitLab**](update/README.md)<br/>Update your GitLab self-managed instance to the latest version. | [**GitLab Releases**](https://about.gitlab.com/releases/)<br/>What's new in GitLab. | ## Popular Documentation @@ -38,7 +38,7 @@ Have a look at some of our most popular documentation resources: | [GitLab CI/CD examples](ci/examples/README.md) | Get up to speed quickly with common CI/CD scenarios. | | [GitLab Container Registry](user/project/container_registry.md) | Host containers within GitLab. | | [GitLab Pages](user/project/pages/index.md) | Host static websites for your projects with GitLab. | -| [GitLab.com settings](user/gitlab_com/index.md) | Settings for [GitLab.com](#gitlabcom). | +| [GitLab.com settings](user/gitlab_com/index.md) | Settings for GitLab.com. | | [Kubernetes integration](user/project/clusters/index.md) | Use GitLab with Kubernetes. | | [SSH authentication](ssh/README.md) | Secure your network communications. | | [Using Docker images](ci/docker/using_docker_images.md) | Build and test your applications with Docker. | @@ -108,7 +108,7 @@ The following documentation relates to the DevOps **Plan** stage: | Plan Topics | Description | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------| | [Burndown Charts](user/project/milestones/burndown_charts.md) **(STARTER)** | Watch your project's progress throughout a specific milestone. | -| [Discussions](user/discussions/index.md) | Threads, comments, and resolvable discussions in issues, commits, and merge requests. | +| [Discussions](user/discussions/index.md) | Threads, comments, and resolvable threads in issues, commits, and merge requests. | | [Due Dates](user/project/issues/due_dates.md) | Keep track of issue deadlines. | | [Epics](user/group/epics/index.md) **(ULTIMATE)** | Tracking groups of issues that share a theme. | | [Issues](user/project/issues/index.md), including [confidential issues](user/project/issues/confidential_issues.md),<br/>[issue and merge request templates](user/project/description_templates.md),<br/>and [moving issues](user/project/issues/managing_issues.md#moving-issues) | Project issues, restricting access to issues, create templates for submitting new issues and merge requests, and moving issues between projects. | @@ -192,7 +192,7 @@ The following documentation relates to the DevOps **Create** stage: |:------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------| | [Checking out merge requests locally](user/project/merge_requests/index.md#checkout-merge-requests-locally) | Tips for working with merge requests locally. | | [Cherry-picking](user/project/merge_requests/cherry_pick_changes.md) | Use GitLab for cherry-picking changes. | -| [Merge request discussion resolution](user/discussions/index.md#moving-a-single-discussion-to-a-new-issue) | Resolve discussions, move discussions in a merge request to an issue, and only allow merge requests to be merged if all discussions are resolved. | +| [Merge request thread resolution](user/discussions/index.md#moving-a-single-thread-to-a-new-issue) | Resolve threads, move threads in a merge request to an issue, and only allow merge requests to be merged if all threads are resolved. | | [Merge requests](user/project/merge_requests/index.md) | Merge request management. | | [Work In Progress "WIP" merge requests](user/project/merge_requests/work_in_progress_merge_requests.md) | Prevent merges of work-in-progress merge requests. | @@ -279,7 +279,7 @@ The following documentation relates to the DevOps **Release** stage: | [Canary Deployments](user/project/canary_deployments.md) **(PREMIUM)** | Employ a popular CI strategy where a small portion of the fleet is updated to the new version first. | | [Deploy Boards](user/project/deploy_boards.md) **(PREMIUM)** | View the current health and status of each CI environment running on Kubernetes, displaying the status of the pods in the deployment. | | [Environments and deployments](ci/environments.md) | With environments, you can control the continuous deployment of your software within GitLab. | -| [Environment-specific variables](ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium) **(PREMIUM)** | Limit scope of variables to specific environments. | +| [Environment-specific variables](ci/variables/README.md#limiting-environment-scopes-of-environment-variables) | Limit scope of variables to specific environments. | | [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Deployment and Delivery with GitLab. | | [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy a static site directly from GitLab. | | [Protected Runners](ci/runners/README.md#protected-runners) | Select Runners to only pick jobs for protected branches and tags. | @@ -302,7 +302,7 @@ The following documentation relates to the DevOps **Configure** stage: | Configure Topics | Description | |:-----------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------| | [Auto DevOps](topics/autodevops/index.md) | Automatically employ a complete DevOps lifecycle. | -| [Easy creation of Kubernetes<br/>clusters on GKE](user/project/clusters/index.md#adding-and-creating-a-new-gke-cluster-via-gitlab) | Use Google Kubernetes Engine and GitLab. | +| [Create Kubernetes clusters on GKE](user/project/clusters/index.md#add-new-gke-cluster) | Use Google Kubernetes Engine and GitLab. | | [Executable Runbooks](user/project/clusters/runbooks/index.md) | Documented procedures that explain how to carry out particular processes. | | [GitLab ChatOps](ci/chatops/README.md) | Interact with CI/CD jobs through chat services. | | [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | @@ -354,97 +354,14 @@ The following documentation relates to the DevOps **Secure** stage: | Secure Topics | Description | |:------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| | [Container Scanning](user/application_security/container_scanning/index.md) **(ULTIMATE)** | Use Clair to scan docker images for known vulnerabilities. | +| [Dependency List](user/application_security/dependency_list/index.md) **(ULTIMATE)** | View your project's dependencies and their known vulnerabilities. | | [Dependency Scanning](user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | | [Dynamic Application Security Testing (DAST)](user/application_security/dast/index.md) **(ULTIMATE)** | Analyze running web applications for known vulnerabilities. | | [Group Security Dashboard](user/application_security/security_dashboard/index.md) **(ULTIMATE)** | View vulnerabilities in all the projects in a group and its subgroups. | -| [License Management](user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | +| [License Compliance](user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | | [Project Security Dashboard](user/application_security/security_dashboard/index.md) **(ULTIMATE)** | View the latest security reports for your project. | | [Static Application Security Testing (SAST)](user/application_security/sast/index.md) **(ULTIMATE)** | Analyze source code for known vulnerabilities. | -## Subscribe to GitLab - -There are two ways to use GitLab: - -- [GitLab self-managed](#gitlab-self-managed): Install, administer, and maintain your own GitLab instance. -- [GitLab.com](#gitlabcom): GitLab's SaaS offering. You don't need to install anything to use GitLab.com, - you only need to [sign up](https://gitlab.com/users/sign_in) and start using GitLab straight away. - -For more information on managing your subscription and [Customers Portal](https://customers.gitlab.com) account, please see [Getting Started with Subscriptions](getting-started/subscription.md). - -The following sections outline tiers and features within GitLab self-managed and GitLab.com. - -<div align="right"> - <a type="button" class="btn btn-default" href="#overview"> - Back to Overview <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### GitLab self-managed - -With GitLab self-managed, you deploy your own GitLab instance on-premises or on a cloud of your choice. -GitLab self-managed is available for [free and with paid subscriptions](https://about.gitlab.com/pricing/#self-managed) in the following tiers: - -| Tier | Includes | -|:---------|:-----------------------------------------------| -| Core | Core features. | -| Starter | Core and Starter features. | -| Premium | Core, Starter, and Premium features. | -| Ultimate | Core, Starter, Premium, and Ultimate features. | - -The following resources are available for more information on GitLab self-managed: - -- [Feature comparison](https://about.gitlab.com/pricing/self-managed/feature-comparison/), for information on what features are available at each tier. -- [GitLab pricing page](https://about.gitlab.com/pricing/#self-managed), for subscription information and a free trial. -- Our [product marketing page](https://about.gitlab.com/handbook/marketing/product-marketing/), for additional information including: - - How [different tiers are licensed](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers). - - The different [GitLab distributions](https://about.gitlab.com/handbook/marketing/product-marketing/#distributions). - -<div align="right"> - <a type="button" class="btn btn-default" href="#overview"> - Back to Overview <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### GitLab.com - -GitLab.com is hosted, managed, and administered by GitLab, Inc., with -[free and paid subscriptions](https://about.gitlab.com/pricing/) for individuals -and teams in the following tiers: - -| Tier | Includes same features available in | -|:-------|:----------------------------------------------------| -| Free | [Core](#gitlab-self-managed) self-managed tier. | -| Bronze | [Starter](#gitlab-self-managed) self-managed tier. | -| Silver | [Premium](#gitlab-self-managed) self-managed tier. | -| Gold | [Ultimate](#gitlab-self-managed) self-managed tier. | - -GitLab.com subscriptions grant access -to the same features available in GitLab self-managed, **except -[administration](administration/index.md) tools and settings**. - -GitLab.com allows you to apply your subscription to a group or your personal user. - -When applied to a **group**, the group, all subgroups, and all projects under the selected group on GitLab.com will have the features of the associated plan. It is recommended to go with a group plan when managing projects and users of an organization. - -When associated with a **personal userspace** instead, all projects will have features with the subscription applied, but as it is not a group, group features will not be available. - -TIP: **Tip:** -To support the open source community and encourage the development of open source projects, GitLab grants access to **Gold** features for all GitLab.com **public** projects, regardless of the subscription. - -The following resources are available for more information on GitLab.com: - -- [Feature comparison](https://about.gitlab.com/pricing/gitlab-com/feature-comparison/), for information on what features are available at each tier. -- [GitLab pricing page](https://about.gitlab.com/pricing/), for subscription information and a free trial. -- Our [product marketing page](https://about.gitlab.com/handbook/marketing/product-marketing/), for additional information including: - - How [different tiers are licensed](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers). - - The different [GitLab distributions](https://about.gitlab.com/handbook/marketing/product-marketing/#distributions). - -<div align="right"> - <a type="button" class="btn btn-default" href="#overview"> - Back to Overview <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## New to Git and GitLab? Working with new systems can be daunting. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index a80ff330e03..8075a40cae7 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -73,6 +73,10 @@ From there, you can see the following actions: - User was added to project and with which [permissions] - Permission changes of a user assigned to a project - User was removed from project +- Project export was downloaded +- Project repository was downloaded +- Project was archived +- Project was unarchived ### Instance events **(PREMIUM ONLY)** @@ -94,6 +98,7 @@ recorded: - Changed password - Ask for password reset - Grant OAuth access +- Started/stopped user impersonation It is possible to filter particular actions by choosing an audit data type from the filter drop-down. You can further filter by specific group, project or user diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 65d36612d85..18c415b5ff7 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -52,7 +52,7 @@ section. **Admin Area > Users**. You will find the option of the access level under the 'Access' section. -  +  1. Click **Save changes** or **Create user** for the changes to take effect. diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index d8094587d14..2fc9db0632e 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -1,19 +1,34 @@ --- comments: false +type: index --- -# Authentication and Authorization +# GitLab authentication and authorization GitLab integrates with the following external authentication and authorization -providers. +providers: -- [LDAP](ldap.md) Includes Active Directory, Apple Open Directory, Open LDAP, - and 389 Server +- [Auth0](../../integration/auth0.md) +- [Authentiq](authentiq.md) +- [Azure](../../integration/azure.md) +- [Bitbucket Cloud](../../integration/bitbucket.md) +- [CAS](../../integration/cas.md) +- [Crowd](../../integration/crowd.md) +- [Facebook](../../integration/facebook.md) +- [GitHub](../../integration/github.md) +- [GitLab.com](../../integration/gitlab.md) +- [Google](../../integration/google.md) +- [JWT](jwt.md) +- [Kerberos](../../integration/kerberos.md) +- [LDAP](ldap.md): Includes Active Directory, Apple Open Directory, Open LDAP, + and 389 Server. - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **(STARTER ONLY)** -- [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, - Bitbucket, Facebook, Shibboleth, Crowd, Azure, Authentiq ID, and JWT -- [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS -- [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider -- [Okta](okta.md) Configure GitLab to sign in using Okta -- [Authentiq](authentiq.md): Enable the Authentiq OmniAuth provider for passwordless authentication -- [Smartcard](smartcard.md) Smartcard authentication **(PREMIUM ONLY)** + - [Google Secure LDAP](google_secure_ldap.md) +- [Okta](okta.md) +- [Salesforce](../../integration/salesforce.md) +- [SAML](../../integration/saml.md) +- [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(SILVER ONLY)** +- [Shibboleth](../../integration/shibboleth.md) +- [Smartcard](smartcard.md) **(PREMIUM ONLY)** +- [Twitter](../../integration/twitter.md) +- [UltraAuth](../../integration/ultra_auth.md) diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 726622d8599..b84eca4ef0d 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Authentiq OmniAuth Provider To enable the Authentiq OmniAuth provider for passwordless authentication you must register an application with Authentiq. @@ -8,47 +12,48 @@ Authentiq will generate a Client ID and the accompanying Client Secret for you t 1. On your GitLab server, open the configuration file: - For omnibus installation - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + For omnibus installation + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - sudo -u git -H editor /home/git/gitlab/config/gitlab.yml - ``` + ```sh + sudo -u git -H editor /home/git/gitlab/config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings to enable single sign-on and add Authentiq as an OAuth provider. 1. Add the provider configuration for Authentiq: - For Omnibus packages: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "authentiq", - "app_id" => "YOUR_CLIENT_ID", - "app_secret" => "YOUR_CLIENT_SECRET", - "args" => { - "scope": 'aq:name email~rs address aq:push' - } - } - ] - ``` - - For installations from source: - - ```yaml - - { name: 'authentiq', - app_id: 'YOUR_CLIENT_ID', - app_secret: 'YOUR_CLIENT_SECRET', - args: { - scope: 'aq:name email~rs address aq:push' - } - } - ``` + For Omnibus packages: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "authentiq", + "app_id" => "YOUR_CLIENT_ID", + "app_secret" => "YOUR_CLIENT_SECRET", + "args" => { + "scope": 'aq:name email~rs address aq:push' + } + } + ] + ``` + + For installations from source: + + ```yaml + - { name: 'authentiq', + app_id: 'YOUR_CLIENT_ID', + app_secret: 'YOUR_CLIENT_SECRET', + args: { + scope: 'aq:name email~rs address aq:push' + } + } + ``` 1. The `scope` is set to request the user's name, email (required and signed), and permission to send push notifications to sign in on subsequent visits. See [OmniAuth Authentiq strategy](https://github.com/AuthentiqID/omniauth-authentiq/wiki/Scopes,-callback-url-configuration-and-responses) for more information on scopes and modifiers. @@ -65,3 +70,15 @@ On the sign in page there should now be an Authentiq icon below the regular sign - If not they will be prompted to download the app and then follow the procedure above. If everything goes right, the user will be returned to GitLab and will be signed in. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 6db74958d5a..ac63b4f2b97 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -1,60 +1,67 @@ +--- +type: reference +--- + # Atlassian Crowd OmniAuth Provider +Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. + ## Configure a new Crowd application 1. Choose 'Applications' in the top menu, then 'Add application'. 1. Go through the 'Add application' steps, entering the appropriate details. The screenshot below shows an example configuration. -  +  ## Configure GitLab 1. On your GitLab server, open the configuration file. - **Omnibus:** + **Omnibus:** - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - **Source:** + **Source:** - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. 1. Add the provider configuration: - **Omnibus:** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "crowd", - "args" => { - "crowd_server_url" => "CROWD_SERVER_URL", - "application_name" => "YOUR_APP_NAME", - "application_password" => "YOUR_APP_PASSWORD" - } - } - ] - ``` - - **Source:** - - ``` - - { name: 'crowd', - args: { - crowd_server_url: 'CROWD_SERVER_URL', - application_name: 'YOUR_APP_NAME', - application_password: 'YOUR_APP_PASSWORD' } } - ``` + **Omnibus:** + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "crowd", + "args" => { + "crowd_server_url" => "CROWD_SERVER_URL", + "application_name" => "YOUR_APP_NAME", + "application_password" => "YOUR_APP_PASSWORD" + } + } + ] + ``` + + **Source:** + + ``` + - { name: 'crowd', + args: { + crowd_server_url: 'CROWD_SERVER_URL', + application_name: 'YOUR_APP_NAME', + application_password: 'YOUR_APP_PASSWORD' } } + ``` + 1. Change `CROWD_SERVER_URL` to the URL of your Crowd server. 1. Change `YOUR_APP_NAME` to the application name from Crowd applications page. 1. Change `YOUR_APP_PASSWORD` to the application password you've set. @@ -77,4 +84,4 @@ could not authorize you from Crowd because invalid credentials Please make sure the Crowd users who need to login to GitLab are authorized to [the application](#configure-a-new-crowd-application) in the step of **Authorisation**. This could be verified by try "Authentication test" for Crowd as of 2.11. -
\ No newline at end of file + diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index 1db5bb4bc3f..55e6f53622c 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Google Secure LDAP **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46391) in GitLab 11.9. @@ -66,7 +70,7 @@ values obtained during the LDAP client configuration earlier: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby + ```ruby gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below main: # 'main' is the GitLab 'provider ID' of this LDAP server @@ -127,7 +131,7 @@ values obtained during the LDAP client configuration earlier: AcZSFJQjdg5BTyvdEDhaYUKGdRw= -----END PRIVATE KEY----- EOS - ``` + ``` 1. Save the file and [reconfigure] GitLab for the changes to take effect. @@ -137,7 +141,7 @@ values obtained during the LDAP client configuration earlier: 1. Edit `config/gitlab.yml`: - ```yaml + ```yaml ldap: enabled: true servers: @@ -204,3 +208,15 @@ values obtained during the LDAP client configuration earlier: [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [restart]: ../restart_gitlab.md#installations-from-source + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 320a65b665d..86dd398343b 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -1,15 +1,9 @@ --- -author: Chris Wilson -author_gitlab: MrChrisW -level: intermediary -article_type: admin guide -date: 2017-05-03 +type: howto --- # How to configure LDAP with GitLab CE -## Introduction - Managing a large number of users in GitLab can become a burden for system administrators. As an organization grows so do user accounts. Keeping these user accounts in sync across multiple enterprise applications often becomes a time consuming task. In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. @@ -268,3 +262,15 @@ have extended functionalities with LDAP, such as: - Multiple LDAP servers Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) **(STARTER ONLY)** for an overview. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md index 2683950f143..366acb9ed3e 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -1,16 +1,10 @@ --- -author: Chris Wilson -author_gitlab: MrChrisW -level: intermediary -article_type: admin guide -date: 2017-05-03 +type: howto --- # How to configure LDAP with GitLab EE **(STARTER ONLY)** -## Introduction - -The present article follows [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. +This article expands on [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. ## GitLab Enterprise Edition - LDAP features @@ -117,3 +111,15 @@ Integration of GitLab with Active Directory (LDAP) reduces the complexity of use It has the advantage of improving user permission controls, whilst easing the deployment of GitLab into an existing [IT environment](https://www.techopedia.com/definition/29199/it-infrastructure). GitLab EE offers advanced group management and multiple LDAP servers. With the assistance of the [GitLab Support](https://about.gitlab.com/support) team, setting up GitLab with an existing AD/LDAP solution will be a smooth and painless process. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_1.png b/doc/administration/auth/img/google_secure_ldap_add_step_1.png Binary files differindex fd254443d75..bee9c602a14 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_1.png +++ b/doc/administration/auth/img/google_secure_ldap_add_step_1.png diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_2.png b/doc/administration/auth/img/google_secure_ldap_add_step_2.png Binary files differindex 611a21ae03c..b127410cb8c 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_2.png +++ b/doc/administration/auth/img/google_secure_ldap_add_step_2.png diff --git a/doc/administration/auth/img/google_secure_ldap_client_settings.png b/doc/administration/auth/img/google_secure_ldap_client_settings.png Binary files differindex 3c0b3f3d4bd..868e6645f56 100644 --- a/doc/administration/auth/img/google_secure_ldap_client_settings.png +++ b/doc/administration/auth/img/google_secure_ldap_client_settings.png diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 497298503ad..e6b3287ce60 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -1,67 +1,71 @@ +--- +type: reference +--- + # JWT OmniAuth provider To enable the JWT OmniAuth provider, you must register your application with JWT. JWT will provide you with a secret key for you to use. -1. On your GitLab server, open the configuration file. - - For Omnibus GitLab: - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - For installations from source: - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` - -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration. - - For Omnibus GitLab: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { name: 'jwt', - args: { - secret: 'YOUR_APP_SECRET', - algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' - uid_claim: 'email', - required_claims: ['name', 'email'], - info_maps: { name: 'name', email: 'email' }, - auth_url: 'https://example.com/', - valid_within: 3600 # 1 hour - } - } - ] - ``` - - For installation from source: - - ``` - - { name: 'jwt', - args: { - secret: 'YOUR_APP_SECRET', - algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' - uid_claim: 'email', - required_claims: ['name', 'email'], - info_map: { name: 'name', email: 'email' }, - auth_url: 'https://example.com/', - valid_within: 3600 # 1 hour - } - } - ``` - - NOTE: **Note:** For more information on each configuration option refer to - the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). - -1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. -1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { name: 'jwt', + args: { + secret: 'YOUR_APP_SECRET', + algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' + uid_claim: 'email', + required_claims: ['name', 'email'], + info_maps: { name: 'name', email: 'email' }, + auth_url: 'https://example.com/', + valid_within: 3600 # 1 hour + } + } + ] + ``` + + For installation from source: + + ``` + - { name: 'jwt', + args: { + secret: 'YOUR_APP_SECRET', + algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' + uid_claim: 'email', + required_claims: ['name', 'email'], + info_map: { name: 'name', email: 'email' }, + auth_url: 'https://example.com/', + valid_within: 3600 # 1 hour + } + } + ``` + + NOTE: **Note:** For more information on each configuration option refer to + the [OmniAuth JWT usage documentation](https://github.com/mbleigh/omniauth-jwt#usage). + +1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. +1. Save the configuration file. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a JWT icon below the regular sign in form. Click the icon to begin the authentication process. JWT will ask the user to @@ -70,3 +74,15 @@ will be redirected to GitLab and will be signed in. [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [restart GitLab]: ../restart_gitlab.md#installations-from-source + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 1a8af0827ee..34f3cfa353f 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -1,36 +1,30 @@ -# LDAP Additions in GitLab EE **(STARTER ONLY)** - -This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP -features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. +--- +type: reference +--- -## Overview +# LDAP Additions in GitLab EE **(STARTER ONLY)** -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which -is a standard application protocol for -accessing and maintaining distributed directory information services -over an Internet Protocol (IP) network. +This section documents LDAP features specific to to GitLab Enterprise Edition +[Starter](https://about.gitlab.com/pricing/#self-managed) and above. -GitLab integrates with LDAP to support **user authentication**. This integration -works with most LDAP-compliant directory servers, including Microsoft Active -Directory, Apple Open Directory, Open LDAP, and 389 Server. -**GitLab Enterprise Edition** includes enhanced integration, including group -membership syncing. +For documentation relevant to both Community Edition and Enterprise Edition, +see the main [LDAP documentation](ldap.md). ## Use cases -- User sync: Once a day, GitLab will update users against LDAP +- User sync: Once a day, GitLab will update users against LDAP. - Group sync: Once an hour, GitLab will update group membership - based on LDAP group members + based on LDAP group members. -## Multiple LDAP servers +## Multiple LDAP servers **(STARTER ONLY)** With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers that your GitLab instance will connect to. -To add another LDAP server, you can start by duplicating the settings under -[the main configuration](ldap.md#configuration) and edit them to match the -additional LDAP server. +To add another LDAP server: + +1. Duplicating the settings under [the main configuration](ldap.md#configuration). +1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. This ID will be stored in the database so that GitLab can remember which LDAP @@ -43,19 +37,19 @@ users against LDAP. The process will execute the following access checks: -1. Ensure the user is still present in LDAP -1. If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This will only be checked if - `active_directory: true` is set in the LDAP configuration [^1] +- Ensure the user is still present in LDAP. +- If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration. [^1] The user will be set to `ldap_blocked` state in GitLab if the above conditions fail. This means the user will not be able to login or push/pull code. The process will also update the following user information: -1. Email address -1. If `sync_ssh_keys` is set, SSH public keys -1. If Kerberos is enabled, Kerberos identity +- Email address. +- If `sync_ssh_keys` is set, SSH public keys. +- If Kerberos is enabled, Kerberos identity. NOTE: **Note:** The LDAP sync process updates existing users while new users will @@ -68,9 +62,6 @@ first time GitLab will trigger a sync for groups the user should be a member of. That way they don't need to wait for the hourly sync to be granted access to their groups and projects. -In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can -also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). - A group sync process will run every hour on the hour, and `group_base` must be set in LDAP configuration for LDAP synchronizations based on group CN to work. This allows GitLab group membership to be automatically updated based on LDAP group members. @@ -85,19 +76,19 @@ following. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - ## - group_base: ou=groups,dc=example,dc=com - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + EOS + ``` 1. [Reconfigure GitLab][reconfigure] for the changes to take effect. @@ -105,24 +96,37 @@ following. 1. Edit `/home/git/gitlab/config/gitlab.yml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` 1. [Restart GitLab][restart] for the changes to take effect. ---- - To take advantage of group sync, group owners or maintainers will need to create an -LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP -groups and/or filters can be linked with a single GitLab group. When the link is -created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, -or Owner). +LDAP group link in their group **Settings > LDAP Groups** page. + +Multiple LDAP groups and [filters](#filters-premium-only) can be linked with +a single GitLab group. When the link is created, an access level/role is +specified (Guest, Reporter, Developer, Maintainer, or Owner). + +### Filters **(PREMIUM ONLY)** + +In GitLab Premium, you can add an LDAP user filter for group synchronization. +Filters allow for complex logic without creating a special LDAP group. + +To sync GitLab group membership based on an LDAP filter: + +1. Open the **LDAP Synchronization** page for the GitLab group. +1. Select **LDAP user filter** as the **Sync method**. +1. Enter an LDAP user filter in the **LDAP user filter** field. + +The filter must comply with the +syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). ## Administrator sync @@ -140,30 +144,30 @@ group, as opposed to the full DN. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - ## - group_base: ou=groups,dc=example,dc=com - - ## - ## The CN of a group containing GitLab administrators - ## - ## Ex. administrators - ## - ## Note: Not `cn=administrators` or the full DN - ## - ## - admin_group: my_admin_group - - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + + ## + ## The CN of a group containing GitLab administrators + ## + ## Ex. administrators + ## + ## Note: Not `cn=administrators` or the full DN + ## + ## + admin_group: my_admin_group + + EOS + ``` 1. [Reconfigure GitLab][reconfigure] for the changes to take effect. @@ -171,26 +175,28 @@ group, as opposed to the full DN. 1. Edit `/home/git/gitlab/config/gitlab.yml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - admin_group: my_admin_group - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` 1. [Restart GitLab][restart] for the changes to take effect. ## Global group memberships lock "Lock memberships to LDAP synchronization" setting allows instance administrators -to lock down user abilities to invite new members to a group. When enabled following happens: +to lock down user abilities to invite new members to a group. -1. Only administrator can manage memberships of any group including access levels. -2. Users are not allowed to share project with other groups or invite members to a project created in a group. +When enabled, the following applies: +- Only administrator can manage memberships of any group including access levels. +- Users are not allowed to share project with other groups or invite members to + a project created in a group. ## Adjusting LDAP user sync schedule @@ -211,9 +217,9 @@ sync to run once every 12 hours at the top of the hour. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" - ``` + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -221,11 +227,11 @@ sync to run once every 12 hours at the top of the hour. 1. Edit `config/gitlab.yaml`: - ```yaml - cron_jobs: - ldap_sync_worker_cron: - "0 */12 * * *" - ``` + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -252,9 +258,9 @@ sync to run once every 2 hours at the top of the hour. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" - ``` + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -262,11 +268,11 @@ sync to run once every 2 hours at the top of the hour. 1. Edit `config/gitlab.yaml`: - ```yaml - cron_jobs: - ldap_group_sync_worker_cron: - "*/30 * * * *" - ``` + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -283,20 +289,20 @@ task. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## An array of CNs of groups containing users that should be considered external - ## - ## Ex. ['interns', 'contractors'] - ## - ## Note: Not `cn=interns` or the full DN - ## - external_groups: ['interns', 'contractors'] - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## An array of CNs of groups containing users that should be considered external + ## + ## Ex. ['interns', 'contractors'] + ## + ## Note: Not `cn=interns` or the full DN + ## + external_groups: ['interns', 'contractors'] + EOS + ``` 1. [Reconfigure GitLab][reconfigure] for the changes to take effect. @@ -304,14 +310,14 @@ task. 1. Edit `config/gitlab.yaml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - external_groups: ['interns', 'contractors'] - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` 1. [Restart GitLab][restart] for the changes to take effect. @@ -330,10 +336,18 @@ administrative duties. ### Supported LDAP group types/attributes -GitLab supports LDAP groups that use member attributes `member`, `submember`, -`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at -least, LDAP groups with object class `groupOfNames`, `posixGroup`, and -`groupOfUniqueName`. Other object classes should work fine as long as members +GitLab supports LDAP groups that use member attributes: + +- `member` +- `submember` +- `uniquemember` +- `memberof` +- `memberuid`. + +This means group sync supports, at least, LDAP groups with object class: +`groupOfNames`, `posixGroup`, and `groupOfUniqueName`. + +Other object classes should work fine as long as members are defined as one of the mentioned attributes. This also means GitLab supports Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. Other LDAP servers should work, too. @@ -341,11 +355,12 @@ Other LDAP servers should work, too. Active Directory also supports nested groups. Group sync will recursively resolve membership if `active_directory: true` is set in the configuration file. -> **Note:** Nested group membership will only be resolved if the nested group - also falls within the configured `group_base`. For example, if GitLab sees a - nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but - the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` - will be ignored. +NOTE: **Note:** +Nested group membership will only be resolved if the nested group +also falls within the configured `group_base`. For example, if GitLab sees a +nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but +the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` +will be ignored. ### Queries @@ -400,7 +415,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server [^1]: In Active Directory, a user is marked as disabled/blocked if the user account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) - has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + has bit 2 set. See <https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/> for more information. ### User DN has changed @@ -420,10 +435,10 @@ things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. This configuration is required for group sync to work properly. - Ensure the correct LDAP group link is added to the GitLab group. Check group - links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. -- Check that the user has an LDAP identity + links by visiting the GitLab group, then **Settings dropdown > LDAP groups**. +- Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area -> Users**. + 1. Navigate to **Admin area > Users**. 1. Search for the user 1. Open the user, by clicking on their name. Do not click 'Edit'. 1. Navigate to the **Identities** tab. There should be an LDAP identity with @@ -434,68 +449,74 @@ Often, the best way to learn more about why group sync is behaving a certain way is to enable debug logging. There is verbose output that details every step of the sync. -1. Start a Rails console +1. Start a Rails console: + + ```bash + # For Omnibus installations + sudo gitlab-rails console - ```bash - # For Omnibus installations - sudo gitlab-rails console + # For installations from source + sudo -u git -H bundle exec rails console production + ``` - # For installations from source - sudo -u git -H bundle exec rails console production - ``` 1. Set the log level to debug (only for this session): - ```ruby - Rails.logger.level = Logger::DEBUG - ``` + ```ruby + Rails.logger.level = Logger::DEBUG + ``` + 1. Choose a GitLab group to test with. This group should have an LDAP group link already configured. If the output is `nil`, the group could not be found. If a bunch of group attributes are output, your group was found successfully. - ```ruby - group = Group.find_by(name: 'my_group') + ```ruby + group = Group.find_by(name: 'my_group') + + # Output + => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> + ``` - # Output - => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> - ``` 1. Run a group sync for this particular group. - ```ruby - EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) - ``` + ```ruby + EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) + ``` + 1. Look through the output of the sync. See [example log output](#example-log-output) below for more information about the output. 1. If you still aren't able to see why the user isn't being added, query the LDAP group directly to see what members are listed. Still in the Rails console, run the following query: - ```ruby - adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider - ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + ```ruby + adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider + ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + + # Output + => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 + ``` - # Output - => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 - ``` 1. Query the LDAP group's member DNs and see if the user's DN is in the list. One of the DNs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, the user does not appear to be in the LDAP group. - ```ruby - ldap_group.member_dns + ```ruby + ldap_group.member_dns + + # Output + => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] + ``` - # Output - => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] - ``` 1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead. If you didn't see results from the last query, try querying by UIDs instead. - ```ruby - ldap_group.member_uids + ```ruby + ldap_group.member_uids - # Output - => ['john','mary'] - ``` + # Output + => ['john','mary'] + ``` #### Example log output @@ -532,8 +553,9 @@ and more DNs may be added, or existing entries modified, based on additional LDAP group lookups. The very last occurrence of this entry should indicate exactly which users GitLab believes should be added to the group. -> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' - and 50 is 'Owner' +NOTE: **Note:** +10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' +and 50 is 'Owner'. ```bash Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 2144f5753a8..186bf4c4825 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -1,30 +1,50 @@ +--- +type: reference +--- + <!-- If the change is EE-specific, put it in `ldap-ee.md`, NOT here. --> # LDAP GitLab integrates with LDAP to support user authentication. -This integration works with most LDAP-compliant directory -servers, including Microsoft Active Directory, Apple Open Directory, Open LDAP, -and 389 Server. GitLab Enterprise Editions include enhanced integration, + +This integration works with most LDAP-compliant directory servers, including: + +- Microsoft Active Directory +- Apple Open Directory +- Open LDAP +- 389 Server. + +GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. -## GitLab EE +For more details about EE-specific LDAP features, see the +[LDAP Enterprise Edition documentation](ldap-ee.md). + +NOTE: **Note:** +The information on this page is relevant for both GitLab CE and EE. + +## Overview -The information on this page is relevant for both GitLab CE and EE. For more -details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](ldap-ee.md). **(STARTER ONLY)** +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which is a standard +application protocol for accessing and maintaining distributed directory +information services over an Internet Protocol (IP) network. ## Security -GitLab assumes that LDAP users are not able to change their LDAP 'mail', 'email' -or 'userPrincipalName' attribute. An LDAP user who is allowed to change their -email on the LDAP server can potentially -[take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) -on your GitLab server. +GitLab assumes that LDAP users: + +- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attribute. + An LDAP user who is allowed to change their email on the LDAP server can potentially + [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) + on your GitLab server. +- Have unique email addresses, otherwise it is possible for LDAP users with the same + email address to share the same GitLab account. We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server. +allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on +the LDAP server or share email addresses. ### User deletion @@ -60,9 +80,12 @@ NOTE: **Note**: In GitLab Enterprise Edition Starter, you can configure multiple LDAP servers to connect to one GitLab server. -For a complete guide on configuring LDAP with GitLab Community Edition, please check -the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). -For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** +For a complete guide on configuring LDAP with: + +- GitLab Community Edition, see + [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +- Enterprise Editions, see + [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -380,7 +403,7 @@ production: Tip: If you want to limit access to the nested members of an Active Directory group, you can use the following syntax: -``` +```text (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` @@ -398,30 +421,30 @@ The `user_filter` DN can contain special characters. For example: - A comma: - ``` - OU=GitLab, Inc,DC=gitlab,DC=com - ``` + ```text + OU=GitLab, Inc,DC=gitlab,DC=com + ``` - Open and close brackets: - ``` - OU=Gitlab (Inc),DC=gitlab,DC=com - ``` + ```text + OU=Gitlab (Inc),DC=gitlab,DC=com + ``` - These characters must be escaped as documented in - [RFC 4515](https://tools.ietf.org/search/rfc4515). + These characters must be escaped as documented in + [RFC 4515](https://tools.ietf.org/search/rfc4515). - Escape commas with `\2C`. For example: - ``` - OU=GitLab\2C Inc,DC=gitlab,DC=com - ``` + ```text + OU=GitLab\2C Inc,DC=gitlab,DC=com + ``` - Escape open and close brackets with `\28` and `\29`, respectively. For example: - ``` - OU=Gitlab \28Inc\29,DC=gitlab,DC=com - ``` + ```text + OU=Gitlab \28Inc\29,DC=gitlab,DC=com + ``` ## Enabling LDAP sign-in for existing GitLab users @@ -445,13 +468,13 @@ the configuration option `lowercase_usernames`. By default, this configuration o 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - # snip... - lowercase_usernames: true - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + lowercase_usernames: true + EOS + ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -459,14 +482,14 @@ the configuration option `lowercase_usernames`. By default, this configuration o 1. Edit `config/gitlab.yaml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - lowercase_usernames: true - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + lowercase_usernames: true + ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -494,13 +517,20 @@ be mandatory and clients cannot be authenticated with the TLS protocol. ## Troubleshooting +If a user account is blocked or unblocked due to the LDAP configuration, a +message will be logged to `application.log`. + +If there is an unexpected error during an LDAP lookup (configuration error, +timeout), the login is rejected and a message will be logged to +`production.log`. + ### Debug LDAP user filter with ldapsearch -This example uses ldapsearch and assumes you are using ActiveDirectory. The +This example uses `ldapsearch` and assumes you are using ActiveDirectory. The following query returns the login names of the users that will be allowed to log in to GitLab if you configure your own user_filter. -``` +```sh ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName ``` @@ -519,26 +549,17 @@ ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$ba - Run the following check command to make sure that the LDAP settings are correct and GitLab can see your users: - ```bash - # For Omnibus installations - sudo gitlab-rake gitlab:ldap:check + ```bash + # For Omnibus installations + sudo gitlab-rake gitlab:ldap:check - # For installations from source - sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production - ``` + # For installations from source + sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production + ``` -### Connection Refused +### Connection refused If you are getting 'Connection Refused' errors when trying to connect to the LDAP server please double-check the LDAP `port` and `encryption` settings used by GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR `encryption: 'simple_tls'` and `port: 636`. - -### Troubleshooting - -If a user account is blocked or unblocked due to the LDAP configuration, a -message will be logged to `application.log`. - -If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the login is rejected and a message will be logged to -`production.log`. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 6e48add6930..3445f916ef4 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # OpenID Connect OmniAuth provider GitLab can use [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) as an OmniAuth provider. @@ -5,101 +9,103 @@ GitLab can use [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0 To enable the OpenID Connect OmniAuth provider, you must register your application with an OpenID Connect provider. The OpenID Connect will provide you with a client details and secret for you to use. -1. On your GitLab server, open the configuration file. - - For Omnibus GitLab: - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - For installations from source: - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` - - See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. - -1. Add the provider configuration. - - For Omnibus GitLab: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { 'name' => 'openid_connect', - 'label' => '<your_oidc_label>', - 'args' => { - "name' => 'openid_connect', - 'scope' => ['openid','profile'], - 'response_type' => 'code', - 'issuer' => '<your_oidc_url>', - 'discovery' => true, - 'client_auth_method' => 'query', - 'uid_field' => '<uid_field>', - 'client_options' => { - 'identifier' => '<your_oidc_client_id>', - 'secret' => '<your_oidc_client_secret>', - 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' - } - } - } - ] - ``` - - For installation from source: - - ```yaml - - { name: 'openid_connect', - label: '<your_oidc_label>', - args: { - name: 'openid_connect', - scope: ['openid','profile'], - response_type: 'code', - issuer: '<your_oidc_url>', - discovery: true, - client_auth_method: 'query', - uid_field: '<uid_field>', - client_options: { - identifier: '<your_oidc_client_id>', - secret: '<your_oidc_client_secret>', - redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback' - } - } - } - ``` - - > **Note:** - > - > - For more information on each configuration option refer to - the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) and - the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + + See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { 'name' => 'openid_connect', + 'label' => '<your_oidc_label>', + 'args' => { + "name' => 'openid_connect', + 'scope' => ['openid','profile'], + 'response_type' => 'code', + 'issuer' => '<your_oidc_url>', + 'discovery' => true, + 'client_auth_method' => 'query', + 'uid_field' => '<uid_field>', + 'client_options' => { + 'identifier' => '<your_oidc_client_id>', + 'secret' => '<your_oidc_client_secret>', + 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ] + ``` + + For installation from source: + + ```yaml + - { name: 'openid_connect', + label: '<your_oidc_label>', + args: { + name: 'openid_connect', + scope: ['openid','profile'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ``` + + NOTE: **Note:** + For more information on each configuration option refer to the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) + and the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). 1. For the configuration above, change the values for the provider to match your OpenID Connect client setup. Use the following as a guide: - - `<your_oidc_label>` is the label that will be displayed on the login page. - - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. - If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. - - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. - - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. - If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. - - `client_options` are the OpenID Connect client-specific options. Specifically: - - - `identifier` is the client identifier as configured in the OpenID Connect service provider. - - `secret` is the client secret as configured in the OpenID Connect service provider. - - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. - - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. - - The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: - - - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. - - `token_endpoint` is the URL to the endpoint that provides Access Token. - - `userinfo_endpoint` is the URL to the endpoint that provides the user information. - - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. - -1. Save the configuration file. -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) - for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + - `<your_oidc_label>` is the label that will be displayed on the login page. + - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. + If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. + - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. + - `client_auth_method` (optional) specifies the method used for authenticating the client with the OpenID Connect provider. + - Supported values are: + - `basic` - HTTP Basic Authentication + - `jwt_bearer` - JWT based authentication (private key and client secret signing) + - `mtls` - Mutual TLS or X.509 certificate validation + - Any other value will POST the client id and secret in the request body + - If not specified, defaults to `basic`. + - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. + If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. + - `client_options` are the OpenID Connect client-specific options. Specifically: + - `identifier` is the client identifier as configured in the OpenID Connect service provider. + - `secret` is the client secret as configured in the OpenID Connect service provider. + - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. + - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. + - The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: + - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. + - `token_endpoint` is the URL to the endpoint that provides Access Token. + - `userinfo_endpoint` is the URL to the endpoint that provides the user information. + - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. + +1. Save the configuration file. +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be an OpenID Connect icon below the regular sign in form. Click the icon to begin the authentication process. The OpenID Connect provider will ask the user to @@ -139,7 +145,7 @@ for more details: } ``` -### Troubleshooting +## Troubleshooting If you're having trouble, here are some tips: @@ -155,9 +161,9 @@ If you're having trouble, here are some tips: `https://accounts.google.com/.well-known/openid-configuration`. 1. The OpenID Connect client uses HTTP Basic Authentication to send the - OAuth2 access token. For example, if you are seeing 401 errors upon - retrieving the `userinfo` endpoint, you may want to check your OpenID - Web server configuration. For example, for + OAuth2 access token if `client_auth_method` is not defined or if set to `basic`. + If you are seeing 401 errors upon retrieving the `userinfo` endpoint, you may + want to check your OpenID Web server configuration. For example, for [oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you may need to [add a configuration parameter to Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index aa4e1b0d2e0..5524c3ba092 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Okta SSO provider Okta is a [Single Sign-on provider](https://www.okta.com/products/single-sign-on/) that can be used to authenticate @@ -16,7 +20,7 @@ The following documentation enables Okta as a SAML provider. 1. Next, you'll need the to fill in the SAML general config. Here's an example image. -  +  1. The last part of the configuration is the feedback section where you can just say you're a customer and creating an app for internal use. @@ -24,7 +28,7 @@ The following documentation enables Okta as a SAML provider. profile. Click on the SAML 2.0 config instructions button which should look like the following: -  +  1. On the screen that comes up take note of the **Identity Provider Single Sign-On URL** which you'll use for the @@ -38,112 +42,112 @@ Now that the Okta app is configured, it's time to enable it in GitLab. ## Configure GitLab -1. On your GitLab server, open the configuration file: - - **For Omnibus GitLab installations** - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - **For installations from source** - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` - -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) - for initial settings. - -1. To allow your users to use Okta to sign up without having to manually create - an account first, don't forget to add the following values to your - configuration: - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` - - **For installations from source** - - ```yaml - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` - -1. You can also automatically link Okta users with existing GitLab users if - their email addresses match by adding the following setting: - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` - - **For installations from source** - - ```yaml - auto_link_saml_user: true - ``` - -1. Add the provider configuration. - - >**Notes:** - > - >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint - of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab - installation to generate the correct value). - > - >- To get the `idp_cert_fingerprint` fingerprint, first download the - certificate from the Okta app you registered and then run: - `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` - with the location of your certificate. - > - >- Change the value of `idp_sso_target_url`, with the value of the - **Identity Provider Single Sign-On URL** from the step when you - configured the Okta app. - > - >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab - to the IdP. - > - >- Leave `name_identifier_format` as-is. - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` - - **For installations from source** - - ```yaml - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ``` +1. On your GitLab server, open the configuration file: + + **For Omnibus GitLab installations** + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + **For installations from source** + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) + for initial settings. + +1. To allow your users to use Okta to sign up without having to manually create + an account first, don't forget to add the following values to your + configuration: + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] + gitlab_rails['omniauth_block_auto_created_users'] = false + ``` + + **For installations from source** + + ```yaml + allow_single_sign_on: ["saml"] + block_auto_created_users: false + ``` + +1. You can also automatically link Okta users with existing GitLab users if + their email addresses match by adding the following setting: + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_auto_link_saml_user'] = true + ``` + + **For installations from source** + + ```yaml + auto_link_saml_user: true + ``` + +1. Add the provider configuration. + + >**Notes:** + > + >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint + of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab + installation to generate the correct value). + > + >- To get the `idp_cert_fingerprint` fingerprint, first download the + certificate from the Okta app you registered and then run: + `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` + with the location of your certificate. + > + >- Change the value of `idp_sso_target_url`, with the value of the + **Identity Provider Single Sign-On URL** from the step when you + configured the Okta app. + > + >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab + to the IdP. + > + >- Leave `name_identifier_format` as-is. + + **For Omnibus GitLab installations** + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Okta' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` + + **For installations from source** + + ```yaml + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + }, + label: 'Okta' # optional label for SAML login button, defaults to "Saml" + } + ``` 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) GitLab for Omnibus and installations from source respectively for the changes to take effect. @@ -157,3 +161,15 @@ Make sure the groups exist and are assigned to the Okta app. You can take a look of the [SAML documentation](../../integration/saml.md#marking-users-as-external-based-on-saml-groups) on external groups since it works the same. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index a0d4e9ef3b5..4f236d1afb8 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Smartcard authentication **(PREMIUM ONLY)** GitLab supports authentication using smartcards. @@ -22,7 +26,7 @@ To use a smartcard with an X.509 certificate to authenticate against a local database with GitLab, `CN` and `emailAddress` must be defined in the certificate. For example: -``` +```text Certificate: Data: Version: 1 (0x0) @@ -56,11 +60,11 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['smartcard_enabled'] = true - gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" - gitlab_rails['smartcard_client_certificate_required_port'] = 3444 - ``` + ```ruby + gitlab_rails['smartcard_enabled'] = true + gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + gitlab_rails['smartcard_client_certificate_required_port'] = 3444 + ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -154,15 +158,15 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - # snip... - # Enable smartcard authentication against the LDAP server. Valid values - # are "false", "optional", and "required". - smartcard_auth: optional - EOS - ``` + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + EOS + ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -171,16 +175,16 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `config/gitlab.yml`: - ```yaml - production: - ldap: - servers: - main: - # snip... - # Enable smartcard authentication against the LDAP server. Valid values - # are "false", "optional", and "required". - smartcard_auth: optional - ``` + ```yaml + production: + ldap: + servers: + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + ``` 1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. @@ -191,9 +195,9 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['smartcard_required_for_git_access'] = true - ``` + ```ruby + gitlab_rails['smartcard_required_for_git_access'] = true + ``` 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -202,13 +206,25 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Edit `config/gitlab.yml`: - ```yaml - ## Smartcard authentication settings - smartcard: - # snip... - # Browser session with smartcard sign-in is required for Git access - required_for_git_access: true - ``` + ```yaml + ## Smartcard authentication settings + smartcard: + # snip... + # Browser session with smartcard sign-in is required for Git access + required_for_git_access: true + ``` 1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 04f52783d22..f5d58db0133 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -27,8 +27,6 @@ but not recommended and out of the scope of this document. Read the [insecure Registry documentation][docker-insecure] if you want to implement this. ---- - **Installations from source** If you have installed GitLab from source: @@ -116,8 +114,6 @@ GitLab from source respectively. Be careful to choose a port different than the one that Registry listens to (`5000` by default), otherwise you will run into conflicts. ---- - **Omnibus GitLab installations** 1. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the @@ -141,7 +137,14 @@ otherwise you will run into conflicts. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- +1. Validate using: + + ```sh + openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:443 > cacert.pem + ``` + +NOTE: **Note:** +If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. **Installations from source** @@ -158,8 +161,6 @@ otherwise you will run into conflicts. 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). ---- - Users should now be able to login to the Container Registry with their GitLab credentials using: @@ -171,14 +172,16 @@ docker login gitlab.example.com:4567 If the Registry is configured to use its own domain, you will need a TLS certificate for that specific domain (e.g., `registry.example.com`) or maybe -a wildcard certificate if hosted under a subdomain of your existing GitLab +a wildcard certificate if hosted under a subdomain of your existing GitLab domain (e.g., `registry.gitlab.example.com`). +NOTE: **Note:** +As well as manually generated SSL certificates (explained here), certificates automatically +generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html#host-services). + Let's assume that you want the container Registry to be accessible at `https://registry.gitlab.example.com`. ---- - **Omnibus GitLab installations** 1. Place your TLS certificate and key in @@ -210,8 +213,6 @@ look like: > registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/certificate.key" > ``` ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and @@ -226,8 +227,6 @@ look like: 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Make the relevant changes in NGINX as well (domain, port, TLS certificates path). ---- - Users should now be able to login to the Container Registry using their GitLab credentials: @@ -252,8 +251,6 @@ Registry application itself. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and @@ -272,8 +269,6 @@ If the Container Registry is enabled, then it will be available on all new projects. To disable this function and let the owners of a project to enable the Container Registry by themselves, follow the steps below. ---- - **Omnibus GitLab installations** 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -284,8 +279,6 @@ the Container Registry by themselves, follow the steps below. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `default_projects_features` @@ -321,8 +314,6 @@ This path is accessible to: > **Warning** You should confirm that all GitLab, Registry and web server users have access to this directory. ---- - **Omnibus GitLab installations** The default location where images are stored in Omnibus, is @@ -336,8 +327,6 @@ The default location where images are stored in Omnibus, is 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** The default location where images are stored in source installations, is @@ -446,8 +435,6 @@ In the examples below we set the Registry's port to `5001`. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** 1. Open the configuration file of your Registry server and edit the @@ -484,7 +471,7 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. ``` 1. A certificate keypair is required for GitLab and the Container Registry to - communicate securely. By default omnibus-gitlab will generate one keypair, + communicate securely. By default Omnibus GitLab will generate one keypair, which is saved to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key`. When using a non-bundled Container Registry, you will need to supply a custom certificate key. To do that, add the following to @@ -500,7 +487,7 @@ You can use GitLab as an auth endpoint and use a non-bundled Container Registry. **Note:** The file specified at `registry_key_path` gets populated with the content specified by `internal_key`, each time reconfigure is executed. If - no file is specified, omnibus-gitlab will default it to + no file is specified, Omnibus GitLab will default it to `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate it. @@ -565,8 +552,6 @@ To configure a notification endpoint in Omnibus: 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** Configuring the notification endpoint is done in your registry config YML file created @@ -640,8 +625,6 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **For installations from source** 1. Edit `config/gitlab.yml`: @@ -673,8 +656,6 @@ You can add a configuration option for backwards compatibility. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **For installations from source** 1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: @@ -696,11 +677,43 @@ project or branch name. Special characters can include: - Trailing hyphen/dash - Double hyphen/dash -To get around this, you can [change the group path](../user/group/index.md#changing-a-groups-path), -[change the project path](../user/project/settings/index.md#renaming-a-repository) or change the -branch name. Another option is to create a [push rule](../push_rules/push_rules.html) to prevent +To get around this, you can [change the group path](../user/group/index.md#changing-a-groups-path), +[change the project path](../user/project/settings/index.md#renaming-a-repository) or change the +branch name. Another option is to create a [push rule](../push_rules/push_rules.html) to prevent this at the instance level. +### Image push errors + +When getting errors or "retrying" loops in an attempt to push an image but `docker login` works fine, +there is likely an issue with the headers forwarded to the registry by NGINX. The default recommended +NGINX configurations should handle this, but it might occur in custom setups where the SSL is +offloaded to a third party reverse proxy. + +This problem was discussed in a [docker project issue][docker-image-push-issue] and a simple solution +would be to enable relative urls in the registry. + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['env'] = { + "REGISTRY_HTTP_RELATIVEURLS" => true + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**For installations from source** + +1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: + + ```yaml + http: + relativeurls: true + ``` + +1. Restart the registry for the changes to take affect. [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates @@ -720,3 +733,4 @@ this at the instance level. [new-domain]: #configure-container-registry-under-its-own-domain [notifications-config]: https://docs.docker.com/registry/notifications/ [registry-notifications-config]: https://docs.docker.com/registry/configuration/#notifications +[docker-image-push-issue]: https://github.com/docker/distribution/issues/970 diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 113514e1ee8..7238d08ab09 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -12,17 +12,17 @@ NOTE: **Note:** Custom Git hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md) Git natively supports hooks that are executed on different actions. -Examples of server-side git hooks include pre-receive, post-receive, and update. +Examples of server-side Git hooks include pre-receive, post-receive, and update. See [Git SCM Server-Side Hooks][hooks] for more information about each hook type. -As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab -administrators can add custom git hooks to any GitLab project. +As of GitLab Shell version 2.2.0 (which requires GitLab 7.5+), GitLab +administrators can add custom Git hooks to any GitLab project. ## Create a custom Git hook for a repository Server-side Git hooks are typically placed in the repository's `hooks` -subdirectory. In GitLab, hook directories are are symlinked to the gitlab-shell -`hooks` directory for ease of maintenance between gitlab-shell upgrades. +subdirectory. In GitLab, hook directories are symlinked to the GitLab Shell +`hooks` directory for ease of maintenance between GitLab Shell upgrades. Custom hooks are implemented differently, but the behavior is exactly the same once the hook is created. Follow the steps below to set up a custom hook for a repository: @@ -36,7 +36,7 @@ repository: 1. Inside the new `custom_hooks` directory, create a file with a name matching the hook type. For a pre-receive hook the file name should be `pre-receive` with no extension. -1. Make the hook file executable and make sure it's owned by git. +1. Make the hook file executable and make sure it's owned by Git. 1. Write the code to make the Git hook function as expected. Hooks can be in any language. Ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will probably be @@ -49,17 +49,17 @@ as appropriate. To create a Git hook that applies to all of your repositories in your instance, set a global Git hook. Since all the repositories' `hooks` -directories are symlinked to gitlab-shell's `hooks` directory, adding any hook -to the gitlab-shell `hooks` directory will also apply it to all repositories. Follow +directories are symlinked to GitLab Shell's `hooks` directory, adding any hook +to the GitLab Shell `hooks` directory will also apply it to all repositories. Follow the steps below to properly set up a custom hook for all repositories: 1. On the GitLab server, navigate to the configured custom hook directory. The - default is in the gitlab-shell directory. The gitlab-shell `hook` directory + default is in the GitLab Shell directory. The GitLab Shell `hook` directory for an installation from source the path is usually `/home/git/gitlab-shell/hooks`. For Omnibus installs the path is usually `/opt/gitlab/embedded/service/gitlab-shell/hooks`. To look in a different directory for the global custom hooks, - set `custom_hooks_dir` in the gitlab-shell config. For + set `custom_hooks_dir` in the GitLab Shell config. For Omnibus installations, this can be set in `gitlab.rb`; and in source installations, this can be set in `gitlab-shell/config.yml`. 1. Create a new directory in this location. Depending on your hook, it will be diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index dc4cc401fca..64eca0b00f6 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -122,6 +122,7 @@ production: discover: nameserver: localhost record: secondary.postgresql.service.consul + record_type: A port: 8600 interval: 60 disconnect_timeout: 120 @@ -137,12 +138,16 @@ The following options can be set: | Option | Description | Default | |----------------------|---------------------------------------------------------------------------------------------------|-----------| | `nameserver` | The nameserver to use for looking up the DNS record. | localhost | -| `record` | The A record to look up. This option is required for service discovery to work. | | +| `record` | The record to look up. This option is required for service discovery to work. | | +| `record_type` | Optional record type to look up, this can be either A or SRV (since GitLab 12.3) | A | | `port` | The port of the nameserver. | 8600 | | `interval` | The minimum time in seconds between checking the DNS record. | 60 | | `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | | `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | +If `record_type` is set to `SRV`, GitLab will continue to use a round-robin algorithm +and will ignore the `weight` and `priority` in the record. + The `interval` value specifies the _minimum_ time between checks. If the A record has a TTL greater than this value, then service discovery will honor said TTL. For example, if the TTL of the A record is 90 seconds, then service diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md index 776c60703fc..5153666705f 100644 --- a/doc/administration/dependency_proxy.md +++ b/doc/administration/dependency_proxy.md @@ -48,7 +48,7 @@ local location or even use object storage. The dependency proxy files for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source -installations under `shared/dependency_proxy/` (relative to the git home directory). +installations under `shared/dependency_proxy/` (relative to the Git home directory). To change the local storage path: **Omnibus GitLab installations** @@ -70,6 +70,7 @@ To change the local storage path: enabled: true storage_path: shared/dependency_proxy ``` + 1. [Restart GitLab] for the changes to take effect. ### Using object storage diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 874b1f3c80d..37d7194af53 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -13,6 +13,7 @@ override certain values. Variable | Type | Description -------- | ---- | ----------- +`ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable) `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (e.g. `//mycdnsubdomain.fictional-cdn.com`) `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 8eee9427b56..27866b7536e 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -171,14 +171,21 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch ## Current limitations -Until [issue #5064][ee-5064] is completed, background verification doesn't cover -CI job artifacts and traces, LFS objects, or user uploads in file storage. -Verify their integrity manually by following [these instructions][foreground-verification] -on both nodes, and comparing the output between them. +Automatic background verification doesn't cover attachments, LFS objects, +job artifacts, and user uploads in file storage. You can keep track of the +progress to include them in [ee-1430]. For now, you can verify their integrity +manually by following [these instructions][foreground-verification] on both +nodes, and comparing the output between them. + +In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects and +archived traces on secondary nodes after the transfer, compares it with the +stored checksums, and rejects transfers if mismatched. Please note that Geo +currently does not support an automatic way to verify these data if they have +been synced before GitLab EE 12.1. Data in object storage is **not verified**, as the object store is responsible for ensuring the integrity of the data. [reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed [foreground-verification]: ../../raketasks/check.md -[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 +[ee-1430]: https://gitlab.com/groups/gitlab-org/-/epics/1430 diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 30d35df25c7..64d7ef2d609 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -30,7 +30,7 @@ To bring the former **primary** node up to date: `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install the GitLab instance from scratch and set it up as a **secondary** node by following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. - + NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) for this node during disaster recovery procedure you may need to [block all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png Binary files differindex fd51523104b..0a7b841897b 100644 --- a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png Binary files differindex b2a6da69d3d..85759d903a4 100644 --- a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index d44e141b66b..ba95843b0b0 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,4 +1,4 @@ -# Disaster Recovery **(PREMIUM ONLY)** +# Disaster Recovery (Geo) **(PREMIUM ONLY)** Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 393326c3347..0c2d48854f0 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -187,7 +187,7 @@ access to the **primary** node during the maintenance window. before it is completed will cause the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: - + - All replication meters to each 100% replicated, 0% failures. - All verification meters reach 100% verified, 0% failures. - Database replication lag is 0ms. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 0e11dffa0d6..fd076bb79d8 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -17,7 +17,7 @@ You are encouraged to first read through all the steps before executing them in your testing/production environment. NOTE: **Note:** -**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. +**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. Any change that requires access to the **Admin Area** needs to be done in the **primary** node because the **secondary** node is a read-only replica. @@ -242,7 +242,7 @@ node's Geo Nodes dashboard in your browser.  If your installation isn't working properly, check the -[troubleshooting document]. +[troubleshooting document](troubleshooting.md). The two most obvious issues that can become apparent in the dashboard are: diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 71a1ce87833..9272287a4a7 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -116,7 +116,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o by default. However, Geo requires the **secondary** node to be able to connect to the **primary** node's database. For this reason, we need the address of each node. - + NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md). If you are using a cloud provider, you can lookup the addresses for each @@ -424,22 +424,22 @@ data before running `pg_basebackup`. - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - If your database is too large to be transferred in 30 minutes, you will need - to increase the timeout, e.g., `--backup-timeout=3600` if you expect the - initial replication to take under an hour. - - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether - (e.g., you know the network path is secure, or you are using a site-to-site - VPN). This is **not** safe over the public Internet! - - You can read more details about each `sslmode` in the - [PostgreSQL documentation][pg-docs-ssl]; - the instructions above are carefully written to ensure protection against - both passive eavesdroppers and active "man-in-the-middle" attackers. - - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the - replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to - add `--force` to the command line. - - When not in a production machine you can disable backup step if you - really sure this is what you want by adding `--skip-backup` + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` The replication process is now complete. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 85687d4a648..256195998a7 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -25,7 +25,6 @@ developed and tested. We aim to be compatible with most external This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. - ### Configure the external database to be replicated To set up an external database, you can either: diff --git a/doc/administration/geo/replication/img/geo_architecture.png b/doc/administration/geo/replication/img/geo_architecture.png Binary files differindex d318cd5d0f4..aac63be41ff 100644 --- a/doc/administration/geo/replication/img/geo_architecture.png +++ b/doc/administration/geo/replication/img/geo_architecture.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index f0d329d5296..dbd466b906d 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,6 +1,12 @@ -# Geo Replication **(PREMIUM ONLY)** +# Replication (Geo) **(PREMIUM ONLY)** -Geo is the solution for widely distributed development teams. +> - Introduced in GitLab Enterprise Edition 8.9. +> - Using Geo in combination with +> [High Availability](../../high_availability/README.md) +> is considered **Generally Available** (GA) in +> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +Replication with Geo is the solution for widely distributed development teams. ## Overview @@ -8,14 +14,8 @@ Fetching large repositories can take a long time for teams located far from a si Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. -> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). -> - Introduced in GitLab Enterprise Edition 8.9. -> - We recommend you use: -> - At least GitLab Enterprise Edition 10.0 for basic Geo features. -> - The latest version for a better experience. -> - Make sure that all nodes run the same GitLab version. -> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). -> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. +NOTE: **Note:** +Check the [requirements](#requirements-for-running-geo) carefully before setting up Geo. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). @@ -111,6 +111,13 @@ The following are required to run Geo: - [Ubuntu](https://www.ubuntu.com) 16.04+ - PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ +- All nodes must run the same GitLab version. + +Additionally, check GitLab's [minimum requirements](../../../install/requirements.md), +and we recommend you use: + +- At least GitLab Enterprise Edition 10.0 for basic Geo features. +- The latest version for a better experience. ### Firewall rules @@ -239,35 +246,48 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. -### Limitations on replication - -Only the following items are replicated to the **secondary** node: - -- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. -- Project repositories. -- Project wiki repositories. -- User uploads. For example, attachments to issues, merge requests, epics, and avatars. -- CI job artifacts and traces. +### Limitations on replication/verification + +The following table lists the GitLab features along with their replication +and verification status on a **secondary** node. + +You can keep track of the progress to include the missing items in: + +- [ee-893](https://gitlab.com/groups/gitlab-org/-/epics/893). +- [ee-1430](https://gitlab.com/groups/gitlab-org/-/epics/1430). + +| Feature | Replicated | Verified | +|-----------|------------|----------| +| All database content (e.g. snippets, epics, issues, merge requests, groups, and project metadata) | Yes | Yes | +| Project repository | Yes | Yes | +| Project wiki repository | Yes | Yes | +| Project designs repository | No | No | +| Uploads (e.g. attachments to issues, merge requests, epics, and avatars) | Yes | Yes, only on transfer, or manually (1) | +| LFS Objects | Yes | Yes, only on transfer, or manually (1) | +| CI job artifacts (other than traces) | Yes | No, only manually (1) | +| Archived traces | Yes | Yes, only on transfer, or manually (1) | +| Personal snippets | Yes | Yes | +| Version-controlled personal snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Project snippets | Yes | Yes | +| Version-controlled project snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Object pools for forked project deduplication | No | No | +| [Server-side Git Hooks](../../custom_hooks.md) | No | No | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | No | No | +| [GitLab Pages](../../pages/index.md) | No | No | +| [Container Registry](../../container_registry.md) ([track progress](https://gitlab.com/gitlab-org/gitlab-ee/issues/2870)) | No | No | +| [NPM Registry](../../npm_registry.md) | No | No | +| [Maven Packages](../../maven_packages.md) | No | No | +| [External merge request diffs](../../merge_request_diffs.md) | No, if they are on-disk | No | +| Content in object storage ([track progress](https://gitlab.com/groups/gitlab-org/-/epics/1526)) | No | No | + +1. The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. DANGER: **DANGER** -Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. - -### Examples of data not replicated - -Take special note that these examples of GitLab features are both: - -- Commonly used. -- **Not** replicated by Geo at present. - -Examples include: - -- [Elasticsearch integration](../../../integration/elasticsearch.md). -- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. -- [GitLab Pages](../../pages/index.md). -- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). - -CAUTION: **Caution:** -If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. +Features not on this list, or with **No** in the **Replicated** column, +are not replicated on the **secondary** node. Failing over without manually +replicating data from those features will cause the data to be **lost**. +If you wish to use those features on a **secondary** node, or to execute a failover +successfully, you must replicate their data using some other means. ## Frequently Asked Questions diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 4f64e21f8ef..e24eb2bd428 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -19,10 +19,10 @@ Once removed from the Geo admin page, you must stop and uninstall the **secondar ```bash # Stop gitlab and remove its supervision process sudo gitlab-ctl uninstall - + # Debian/Ubuntu sudo dpkg --remove gitlab-ee - + # Redhat/Centos sudo rpm --erase gitlab-ee ``` @@ -32,9 +32,9 @@ Once GitLab has been uninstalled from the **secondary** node, the replication sl 1. On the **primary** node, start a PostgreSQL console session: ```bash - sudo gitlab-psql + sudo gitlab-psql ``` - + NOTE: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. @@ -43,9 +43,9 @@ Once GitLab has been uninstalled from the **secondary** node, the replication sl ```sql SELECT * FROM pg_replication_slots; ``` - + 1. Remove the replication slot for the **secondary** node: ```sql SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` + ``` diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 28abfff973d..fe1557fd8b5 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -230,7 +230,7 @@ sudo gitlab-ctl reconfigure This will increase the timeout to three hours (10800 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. -### Reseting Geo **secondary** node replication +### Resetting Geo **secondary** node replication If you get a **secondary** node in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you: @@ -524,6 +524,20 @@ If it doesn't exist or inadvertent changes have been made to it, run `sudo gitla If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. +### An existing tracking database cannot be reused + +Geo cannot reuse an existing tracking database. + +It is safest to use a fresh secondary, or reset the whole secondary by following +[Resetting Geo secondary node replication](#resetting-geo-secondary-node-replication). + +If you are not concerned about possible orphaned directories and files, then you +can simply reset the existing tracking database with: + +```sh +sudo gitlab-rake geo:db:reset +``` + ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node. This error refers to a problem with the database replica on a **secondary** node, diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index c27f6c78455..39174780e24 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -10,10 +10,33 @@ all you need to do is update GitLab itself: 1. Log into each node (**primary** and **secondary** nodes). 1. [Update GitLab][update]. -1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when - the tracking database is enabled. 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. +### Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + +## Upgrading to GitLab 12.1 + +By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure. + +This can be temporarily disabled by running the following before ugprading: + +```sh +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + ## Upgrading to GitLab 10.8 Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node: @@ -241,7 +264,7 @@ Omnibus is the following: 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. 1. Install GitLab on the **secondary** server. -1. Re-run the [database replication process][database-replication]. +1. Re-run the [database replication process](database.md#step-3-initiate-the-replication-process). ## Special update notes for 9.0.x @@ -409,22 +432,7 @@ is prepended with the relevant node for better clarity: sudo gitlab-ctl start ``` -## Check status after updating - -Now that the update process is complete, you may want to check whether -everything is working correctly: - -1. Run the Geo raketask on all nodes, everything should be green: - - ```sh - sudo gitlab-rake gitlab:geo:check - ``` - -1. Check the **primary** node's Geo dashboard for any errors. -1. Test the data replication by pushing code to the **primary** node and see if it - is received by **secondary** nodes. - -## Update tracking database on **secondary** node +### Update tracking database on **secondary** node After updating a **secondary** node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 1780e1babe9..9d653d4e09e 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -53,7 +53,7 @@ sudo service ssh restart ## Instructions In order to use the new protocol, clients need to either pass the configuration -`-c protocol.version=2` to the git command, or set it globally: +`-c protocol.version=2` to the Git command, or set it globally: ```sh git config --global protocol.version 2 diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 4407facfca9..daec5f14e36 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,127 +2,127 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to Git repositories. Without it, no other -components can read or write Git data. +components can read or write Git data. GitLab components that access Git +repositories (gitlab-rails, gitlab-shell, gitlab-workhorse, etc.) act as clients +to Gitaly. End users do not have direct access to Gitaly. -GitLab components that access Git repositories (gitlab-rails, -gitlab-shell, gitlab-workhorse) act as clients to Gitaly. End users do -not have direct access to Gitaly. +In the rest of this page, Gitaly server is referred to the standalone node that +only runs Gitaly, and Gitaly client to the GitLab Rails node that runs all other +processes except Gitaly. ## Configuring Gitaly The Gitaly service itself is configured via a TOML configuration file. -This file is documented [in the gitaly +This file is documented [in the Gitaly repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md). -To change a Gitaly setting in Omnibus you can use -`gitaly['my_setting']` in `/etc/gitlab/gitlab.rb`. Changes will be applied -when you run `gitlab-ctl reconfigure`. +In case you want to change some of its settings: -```ruby -gitaly['prometheus_listen_addr'] = 'localhost:9236' -``` +**For Omnibus GitLab** -To change a Gitaly setting in installations from source you can edit -`/home/git/gitaly/config.toml`. Changes will be applied when you run -`service gitlab restart`. +1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -```toml -prometheus_listen_addr = "localhost:9236" -``` +**For installations from source** -## Client-side GRPC logs - -Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when -you are seeing Gitaly errors. You can control the log level of the -gRPC client with the `GRPC_LOG_LEVEL` environment variable. The -default level is `WARN`. +1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## Running Gitaly on its own server -> This is an optional way to deploy Gitaly which can benefit GitLab +This is an optional way to deploy Gitaly which can benefit GitLab installations that are larger than a single machine. Most installations will be better served with the default configuration used by Omnibus and the GitLab source installation guide. Starting with GitLab 11.4, Gitaly is able to serve all Git requests without -needed a shared NFS mount for Git repository data. +requiring a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still be leveraged for redudancy on block level of the Git data. But only has to be mounted on the Gitaly server. -NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend -using EFS as it may impact GitLab's performance. Please review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) +Starting with GitLab 11.8, it is possible to use ElasticSearch in conjunction with +a Gitaly setup that isn't utilising NFS. In order to use ElasticSearch in this +scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) +needs to be enabled in your GitLab configuration. + +NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended +to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. ### Network architecture -- gitlab-rails shards repositories into "repository storages" -- `gitlab-rails/config/gitlab.yml` contains a map from storage names to - (Gitaly address, Gitaly token) pairs -- the `storage name` -\> `(Gitaly address, Gitaly token)` map in - `gitlab.yml` is the single source of truth for the Gitaly network - topology -- a (Gitaly address, Gitaly token) corresponds to a Gitaly server -- a Gitaly server hosts one or more storages -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients -- Gitaly clients are: unicorn, sidekiq, gitlab-workhorse, - gitlab-shell, Elasticsearch Indexer, and Gitaly itself -- special case: a Gitaly server must be able to make RPC calls **to - itself** via its own (Gitaly address, Gitaly token) pair as - specified in `gitlab-rails/config/gitlab.yml` -- Gitaly servers must not be exposed to the public internet +The following list depicts what the network architecture of Gitaly is: -Gitaly network traffic is unencrypted by default, but supports -[TLS](#tls-support). Authentication is done through a static token. +- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). +- `/config/gitlab.yml` contains a map from storage names to + `(Gitaly address, Gitaly token)` pairs. +- the `storage name` -\> `(Gitaly address, Gitaly token)` map in + `/config/gitlab.yml` is the single source of truth for the Gitaly network + topology. +- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. +- A Gitaly server hosts one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly clients are: Unicorn, Sidekiq, gitlab-workhorse, + gitlab-shell, Elasticsearch Indexer, and Gitaly itself. +- A Gitaly server must be able to make RPC calls **to itself** via its own + `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. +- Gitaly servers must not be exposed to the public internet as Gitaly's network + traffic is unencrypted by default. The use of firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#tls-support). +- Authentication is done through a static token which is shared among the Gitaly + and GitLab Rails nodes. -NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to -restrict access to your Gitaly server. +Below we describe how to configure two Gitaly servers one at +`gitaly1.internal` and the other at `gitaly2.internal` +with secret token `abc123secret`. We assume +your GitLab installation has three repository storages: `default`, +`storage1` and `storage2`. -Below we describe how to configure a Gitaly server at address -`gitaly.internal:8075` with secret token `abc123secret`. We assume -your GitLab installation has two repository storages, `default` and -`storage1`. +### 1. Installation -### Installation +First install Gitaly on each Gitaly server using either +Omnibus GitLab or install it from source: -First install Gitaly using either Omnibus or from source. +- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **_do not_** provide the `EXTERNAL_URL=` value. +- From source: [Install Gitaly](../../install/installation.md#install-gitaly). -Omnibus: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab -package you want using **steps 1 and 2** from the GitLab downloads page but -**_do not_** provide the `EXTERNAL_URL=` value. +### 2. Client side token configuration -Source: [Install Gitaly](../../install/installation.md#install-gitaly) +Configure a token on the instance that runs the GitLab Rails application. -### Client side token configuration +**For Omnibus GitLab** -Configure a token on the client side. +1. On the client node(s), edit `/etc/gitlab/gitlab.rb`: -Omnibus installations: + ```ruby + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -```ruby -# /etc/gitlab/gitlab.rb -gitlab_rails['gitaly_token'] = 'abc123secret' -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -Source installations: +**For installations from source** -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - gitaly: - token: 'abc123secret' -``` +1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + gitlab: + gitaly: + token: 'abc123secret' + ``` -You need to reconfigure (Omnibus) or restart (source) for these -changes to be picked up. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -### Gitaly server configuration +### 3. Gitaly server configuration -Next, on the Gitaly server, we need to configure storage paths, enable +Next, on the Gitaly servers, you need to configure storage paths, enable the network listener and configure the token. NOTE: **Note:** if you want to reduce the risk of downtime when you enable @@ -137,131 +137,228 @@ the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gi from an existing GitLab server to the Gitaly server. Without this shared secret, Git operations in GitLab will result in an API error. -NOTE: **Note:** In most or all cases the storage paths below end in `/repositories` which is -different than `path` in `git_data_dirs` of Omnibus installations. Check the -directory layout on your Gitaly server to be sure. +NOTE: **Note:** +In most or all cases, the storage paths below end in `/repositories` which is +not the case with `path` in `git_data_dirs` of Omnibus GitLab installations. +Check the directory layout on your Gitaly server to be sure. + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + <!-- + updates to following example must also be made at + https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + # /etc/gitlab/gitlab.rb + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['auth_token'] = 'abc123secret' + + # To use TLS for Gitaly you need to add + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + + For `gitaly1.internal`: + + ``` + gitaly['storage'] = [ + { 'name' => 'default' }, + { 'name' => 'storage1' }, + ] + ``` + + For `gitaly2.internal`: + + ``` + gitaly['storage'] = [ + { 'name' => 'storage2' }, + ] + ``` + + NOTE: **Note:** + In some cases, you'll have to set `path` for `gitaly['storage']` in the + format `'path' => '/mnt/gitlab/<storage name>/repositories'`. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. On the client node(s), edit `/home/git/gitaly/config.toml`: + + ```toml + listen_addr = '0.0.0.0:8075' + tls_listen_addr = '0.0.0.0:9999' + + [tls] + certificate_path = /path/to/cert.pem + key_path = /path/to/key.pem + + [auth] + token = 'abc123secret' + ``` + +1. Append the following to `/home/git/gitaly/config.toml` for each respective server: + + For `gitaly1.internal`: + + ```toml + [[storage]] + name = 'default' + + [[storage]] + name = 'storage1' + ``` + + For `gitaly2.internal`: + + ```toml + [[storage]] + name = 'storage2' + ``` + + NOTE: **Note:** + In some cases, you'll have to set `path` for each `[[storage]]` in the + format `path = '/mnt/gitlab/<storage name>/repositories'`. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +### 4. Converting clients to use the Gitaly server + +As the final step, you need to update the client machines to switch from using +their local Gitaly service to the new Gitaly server you just configured. This +is a risky step because if there is any sort of network, firewall, or name +resolution problem preventing your GitLab server from reaching the Gitaly server, +then all Gitaly requests will fail. -Omnibus installations: +Additionally, you need to +[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). -<!-- -updates to following example must also be made at -https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab ---> +We assume that your `gitaly1.internal` Gitaly server can be reached at +`gitaly1.internal:8075` from your GitLab server, and that Gitaly server +can read and write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. -```ruby -# /etc/gitlab/gitlab.rb - -# Avoid running unnecessary services on the Gitaly server -postgresql['enable'] = false -redis['enable'] = false -nginx['enable'] = false -prometheus['enable'] = false -unicorn['enable'] = false -sidekiq['enable'] = false -gitlab_workhorse['enable'] = false - -# Prevent database connections during 'gitlab-ctl reconfigure' -gitlab_rails['rake_cache_clear'] = false -gitlab_rails['auto_migrate'] = false - -# Configure the gitlab-shell API callback URL. Without this, `git push` will -# fail. This can be your 'front door' GitLab URL or an internal load -# balancer. -# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. -gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - -# Make Gitaly accept connections on all network interfaces. You must use -# firewalls to restrict access to this address/port. -gitaly['listen_addr'] = "0.0.0.0:8075" -gitaly['auth_token'] = 'abc123secret' - -gitaly['storage'] = [ - { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' }, - { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' }, -] +We assume also that your `gitaly2.internal` Gitaly server can be reached at +`gitaly2.internal:8075` from your GitLab server, and that Gitaly server +can read and write to `/mnt/gitlab/storage2`. -# To use TLS for Gitaly you need to add -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "path/to/cert.pem" -gitaly['key_path'] = "path/to/key.pem" -``` +**For Omnibus GitLab** -Source installations: +1. Edit `/etc/gitlab/gitlab.rb`: -```toml -# /home/git/gitaly/config.toml -listen_addr = '0.0.0.0:8075' -tls_listen_addr = '0.0.0.0:9999' + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) -[tls] -certificate_path = /path/to/cert.pem -key_path = /path/to/key.pem + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -[auth] -token = 'abc123secret' +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Tail the logs to see the requests: -[[storage]] -name = 'default' -path = '/mnt/gitlab/default/repositories' + ```sh + sudo gitlab-ctl tail gitaly + ``` -[[storage]] -name = 'storage1' -path = '/mnt/gitlab/storage1/repositories' -``` +**For installations from source** -Again, reconfigure (Omnibus) or restart (source). +1. Edit `/home/git/gitlab/config/gitlab.yml`: -### Converting clients to use the Gitaly server + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tcp://gitaly1.internal:8075 + path: /some/dummy/path + storage1: + gitaly_address: tcp://gitaly1.internal:8075 + path: /some/dummy/path + storage2: + gitaly_address: tcp://gitaly2.internal:8075 + path: /some/dummy/path -Now as the final step update the client machines to switch from using -their local Gitaly service to the new Gitaly server you just -configured. This is a risky step because if there is any sort of -network, firewall, or name resolution problem preventing your GitLab -server from reaching the Gitaly server then all Gitaly requests will -fail. + gitaly: + token: 'abc123secret' + ``` -Additionally, you need to -[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). + NOTE: **Note:** + `/some/dummy/path` should be set to a local folder that exists, however no + data will be stored in this folder. This will no longer be necessary after + [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. -We assume that your Gitaly server can be reached at -`gitaly.internal:8075` from your GitLab server, and that Gitaly can read and -write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Tail the logs to see the requests: -Omnibus installations: + ```sh + tail -f /home/git/gitlab/log/gitaly.log + ``` -```ruby -# /etc/gitlab/gitlab.rb -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, -}) +When you tail the Gitaly logs on your Gitaly server you should see requests +coming in. One sure way to trigger a Gitaly request is to clone a repository +from your GitLab server over HTTP. -gitlab_rails['gitaly_token'] = 'abc123secret' -``` +### Disabling the Gitaly service in a cluster environment -Source installations: - -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - repositories: - storages: - default: - path: /mnt/gitlab/default/repositories - gitaly_address: tcp://gitaly.internal:8075 - storage1: - path: /mnt/gitlab/storage1/repositories - gitaly_address: tcp://gitaly.internal:8075 - - gitaly: - token: 'abc123secret' -``` +If you are running Gitaly [as a remote +service](#running-gitaly-on-its-own-server) you may want to disable +the local Gitaly service that runs on your GitLab server by default. +Disabling Gitaly only makes sense when you run GitLab in a custom +cluster configuration, where different services run on different +machines. Disabling Gitaly on all machines in the cluster is not a +valid configuration. + +To disable Gitaly on a client node: + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['enable'] = false + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Edit `/etc/default/gitlab`: -Now reconfigure (Omnibus) or restart (source). When you tail the -Gitaly logs on your Gitaly server (`sudo gitlab-ctl tail gitaly` or -`tail -f /home/git/gitlab/log/gitaly.log`) you should see requests -coming in. One sure way to trigger a Gitaly request is to clone a -repository from your GitLab server over HTTP. + ```shell + gitaly_enabled=false + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). ## TLS support @@ -271,188 +368,209 @@ Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. -The admin needs to bring their own certificate as we do not provide that automatically. -The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +You will need to bring your own certificates as this isn't provided automatically. +The certificate to be used needs to be installed on all Gitaly nodes and on all +client nodes that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -Note that it is possible to configure Gitaly servers with both an +NOTE: **Note:** +It is possible to configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to do a gradual transition from unencrypted to encrypted traffic, if necessary. -To observe what type of connections are actually being used in a -production environment you can use the following Prometheus query: +To configure Gitaly with TLS: -``` -sum(rate(gitaly_connections_total[5m])) by (type) -``` +**For Omnibus GitLab** -### Example TLS configuration +1. On the client nodes, edit `/etc/gitlab/gitlab.rb`: -### Omnibus installations: + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) -#### On client nodes: + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` -```ruby -# /etc/gitlab/gitlab.rb -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly.internal:9999' }, -}) +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly server nodes, edit `/etc/gitlab/gitlab.rb`: -gitlab_rails['gitaly_token'] = 'abc123secret' -``` + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` -#### On Gitaly server nodes: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -```ruby -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "path/to/cert.pem" -gitaly['key_path'] = "path/to/key.pem" -``` +**For installations from source** -### Source installations: +1. On the client nodes, edit `/home/git/gitlab/config/gitlab.yml`: -#### On client nodes: + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tls://gitaly1.internal:9999 + path: /some/dummy/path + storage1: + gitaly_address: tls://gitaly1.internal:9999 + path: /some/dummy/path + storage2: + gitaly_address: tls://gitaly2.internal:9999 + path: /some/dummy/path -```yaml -# /home/git/gitlab/config/gitlab.yml -gitlab: - repositories: - storages: - default: - path: /mnt/gitlab/default/repositories - gitaly_address: tls://gitaly.internal:9999 - storage1: - path: /mnt/gitlab/storage1/repositories - gitaly_address: tls://gitaly.internal:9999 + gitaly: + token: 'abc123secret' + ``` - gitaly: - token: 'abc123secret' -``` + NOTE: **Note:** + `/some/dummy/path` should be set to a local folder that exists, however no + data will be stored in this folder. This will no longer be necessary after + [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`: + + ```toml + tls_listen_addr = '0.0.0.0:9999' + + [tls] + certificate_path = '/path/to/cert.pem' + key_path = '/path/to/key.pem' + ``` -#### On Gitaly server nodes: +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -```toml -# /home/git/gitaly/config.toml -tls_listen_addr = '0.0.0.0:9999' +To observe what type of connections are actually being used in a +production environment you can use the following Prometheus query: -[tls] -certificate_path = '/path/to/cert.pem' -key_path = '/path/to/key.pem' +``` +sum(rate(gitaly_connections_total[5m])) by (type) ``` -## Gitaly-ruby +## `gitaly-ruby` -Gitaly was developed to replace Ruby application code in gitlab-ce/ee. +Gitaly was developed to replace the Ruby application code in GitLab. In order to save time and/or avoid the risk of rewriting existing application logic, in some cases we chose to copy some application code -from gitlab-ce into Gitaly almost as-is. To be able to run that code, we -made gitaly-ruby, which is a sidecar process for the main Gitaly Go -process. Some examples of things that are implemented in gitaly-ruby are -RPC's that deal with wiki's, and RPC's that create commits on behalf of +from GitLab into Gitaly almost as-is. To be able to run that code, +`gitaly-ruby` was created, which is a "sidecar" process for the main Gitaly Go +process. Some examples of things that are implemented in `gitaly-ruby` are +RPCs that deal with wikis, and RPCs that create commits on behalf of a user, such as merge commits. -### Number of gitaly-ruby workers +### Number of `gitaly-ruby` workers -Gitaly-ruby has much less capacity than Gitaly itself. If your Gitaly -server has to handle a lot of request, the default setting of having -just 1 active gitaly-ruby sidecar might not be enough. If you see -ResourceExhausted errors from Gitaly it's very likely that you have not -enough gitaly-ruby capacity. +`gitaly-ruby` has much less capacity than Gitaly itself. If your Gitaly +server has to handle a lot of requests, the default setting of having +just one active `gitaly-ruby` sidecar might not be enough. If you see +`ResourceExhausted` errors from Gitaly, it's very likely that you have not +enough `gitaly-ruby` capacity. -You can increase the number of gitaly-ruby processes on your Gitaly +You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following settings. -Omnibus: +**For Omnibus GitLab** -```ruby -# /etc/gitlab/gitlab.rb -# Default is 2 workers. The minimum is 2; 1 worker is always reserved as -# a passive stand-by. -gitaly['ruby_num_workers'] = 4 -``` +1. Edit `/etc/gitlab/gitlab.rb`: -Source: + ```ruby + # Default is 2 workers. The minimum is 2; 1 worker is always reserved as + # a passive stand-by. + gitaly['ruby_num_workers'] = 4 + ``` -```toml -# /home/git/gitaly/config.toml -[gitaly-ruby] -num_workers = 4 -``` +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Observing gitaly-ruby traffic +**For installations from source** -Gitaly-ruby is a somewhat hidden, internal implementation detail of -Gitaly. There is not that much visibility into what goes on inside -gitaly-ruby processes. +1. Edit `/home/git/gitaly/config.toml`: -If you have Prometheus set up to scrape your Gitaly process, you can see -request rates and error codes for individual RPC's in gitaly-ruby by -querying `grpc_client_handled_total`. Strictly speaking this metric does -not differentiate between gitaly-ruby and other RPC's, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its gitaly-ruby sidecars. + ```toml + [gitaly-ruby] + num_workers = 4 + ``` -Assuming your `grpc_client_handled_total` counter only observes Gitaly, -the following query shows you RPC's are (most likely) internally -implemented as calls to gitaly-ruby. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -``` -sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 -``` +## Eliminating NFS altogether -## Disabling or enabling the Gitaly service in a cluster environment +If you are planning to use Gitaly without NFS for your storage needs +and want to eliminate NFS from your environment altogether, there are +a few things that you need to do: -If you are running Gitaly [as a remote -service](#running-gitaly-on-its-own-server) you may want to disable -the local Gitaly service that runs on your GitLab server by default. +1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. +1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md) + to eliminate the need for a shared authorized_keys file. +1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [live tracing](../job_traces.md#new-live-trace-architecture). +1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). + +NOTE: **Note:** +One current feature of GitLab that still requires a shared directory (NFS) is +[GitLab Pages](../../user/project/pages/index.md). +There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) +to eliminate the need for NFS to support GitLab Pages. -> 'Disabling Gitaly' only makes sense when you run GitLab in a custom -cluster configuration, where different services run on different -machines. Disabling Gitaly on all machines in the cluster is not a -valid configuration. +## Limiting RPC concurrency -If you are setting up a GitLab cluster where Gitaly does not need to -run on all machines, you can disable the Gitaly service in your -Omnibus installation, add the following line to `/etc/gitlab/gitlab.rb`: +It can happen that CI clone traffic puts a large strain on your Gitaly +service. The bulk of the work gets done in the SSHUploadPack (for Git +SSH) and PostUploadPack (for Git HTTP) RPC's. To prevent such workloads +from overcrowding your Gitaly server you can set concurrency limits in +Gitaly's configuration file. ```ruby -gitaly['enable'] = false +# in /etc/gitlab/gitlab.rb + +gitaly['concurrency'] = [ + { + 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", + 'max_per_repo' => 20 + }, + { + 'rpc' => "/gitaly.SSHService/SSHUploadPack", + 'max_per_repo' => 20 + } +] ``` +This will limit the number of in-flight RPC calls for the given RPC's. +The limit is applied per repository. In the example above, each on the +Gitaly server can have at most 20 simultaneous PostUploadPack calls in +flight, and the same for SSHUploadPack. If another request comes in for +a repository that hase used up its 20 slots, that request will get +queued. -When you run `gitlab-ctl reconfigure` the Gitaly service will be -disabled. - -To disable the Gitaly service in a GitLab cluster where you installed -GitLab from source, add the following to `/etc/default/gitlab` on the -machine where you want to disable Gitaly. - -```shell -gitaly_enabled=false -``` +You can observe the behavior of this queue via the Gitaly logs and via +Prometheus. In the Gitaly logs, you can look for the string (or +structured log field) `acquire_ms`. Messages that have this field are +reporting about the concurrency limiter. In Prometheus, look for the +`gitaly_rate_limiting_in_progress`, `gitaly_rate_limiting_queued` and +`gitaly_rate_limiting_seconds` metrics. -When you run `service gitlab restart` Gitaly will be disabled on this -particular machine. +The name of the Prometheus metric is not quite right because this is a +concurrency limiter, not a rate limiter. If a client makes 1000 requests +in a row in a very short timespan, the concurrency will not exceed 1, +and this mechanism (the concurrency limiter) will do nothing. -## Eliminating NFS altogether +## Troubleshooting Gitaly -If you are planning to use Gitaly without NFS for your storage needs -and want to eliminate NFS from your environment altogether, there are -a few things that you need to do: +### Commits, pushes, and clones return a 401 - 1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. - 1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md) - to eliminate the need for a shared authorized_keys file. - 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [live tracing](../job_traces.md#new-live-trace-architecture). - 1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). - 1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +``` +remote: GitLab: 401 Unauthorized +``` -NOTE: **Note:** One current feature of GitLab still requires a shared directory (NFS): [GitLab Pages](../../user/project/pages/index.md). -There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) -to eliminate the need for NFS to support GitLab Pages. +You will need to sync your `gitlab-secrets.json` file with your GitLab +app nodes. -## Troubleshooting Gitaly in production +### `gitaly-debug` Since GitLab 11.6, Gitaly comes with a command-line tool called `gitaly-debug` that can be run on a Gitaly server to aid in @@ -462,3 +580,153 @@ Git clone speed for a specific repository on the server. For an up to date list of sub-commands see [the gitaly-debug README](https://gitlab.com/gitlab-org/gitaly/blob/master/cmd/gitaly-debug/README.md). + +### Client side GRPC logs + +Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC +client has its own log file which may contain useful information when +you are seeing Gitaly errors. You can control the log level of the +gRPC client with the `GRPC_LOG_LEVEL` environment variable. The +default level is `WARN`. + +### Observing `gitaly-ruby` traffic + +[`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly, +so, there's not that much visibility into what goes on inside +`gitaly-ruby` processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPCs in `gitaly-ruby` by +querying `grpc_client_handled_total`. Strictly speaking, this metric does +not differentiate between `gitaly-ruby` and other RPCs, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPCs are (most likely) internally +implemented as calls to `gitaly-ruby`: + +``` +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```sh + remote: GitLab: 401 Unauthorized + To <REMOTE_URL> + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '<REMOTE_URL>' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediatley fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.md#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```sh + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Command line tools cannot connect to Gitaly + +If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. + +Verify that you can reach Gitaly via TCP: + +```bash +sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] +``` + +If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. + +If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. + +If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: + +```bash +echo $http_proxy +echo $https_proxy +``` + +If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. + +To remove the proxy setting, run the following commands (depending on which variables had values): + +```bash +unset http_proxy +unset https_proxy +```
\ No newline at end of file diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index e81d2741082..56665ba8b9a 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -1,3 +1,7 @@ +--- +type: reference, concepts +--- + # Scaling and High Availability GitLab supports several different types of clustering and high-availability. @@ -160,25 +164,24 @@ contention due to certain workloads. #### Reference Architecture -- **Status:** Work-in-progress - **Supported Users (approximate):** 10,000 -- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513), - [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110) - -The Support and Quality teams are in the process of building and performance testing -an environment that will support about 10,000 users. The specifications below -are a work-in-progress representation of the work so far. Quality will be -certifying this environment in FY20-Q2. The specifications may be adjusted -prior to certification based on performance testing. - -- 3 PostgreSQL - 4 CPU, 8GB RAM per node -- 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM per node -- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node -- 4 Sidekiq - 4 CPU, 8GB RAM per node -- 5 GitLab application nodes - 20 CPU, 64GB RAM per node -- 1 Gitaly - 20 CPU, 64GB RAM -- 1 Monitoring node - 4 CPU, 8GB RAM +- **Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [gitlab-org/gitlab-ce/issues/64335](https://gitlab.com/gitlab-org/gitlab-ce/issues/64335) + +The Support and Quality teams built, performance tested, and validated an +environment that supports about 10,000 users. The specifications below are a +representation of the work so far. The specifications may be adjusted in the +future based on additional testing and iteration. + +NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + +- 3 PostgreSQL - 4 CPU, 16GiB memory per node +- 1 PgBouncer - 2 CPU, 4GiB memory +- 2 Redis - 2 CPU, 8GiB memory per node +- 3 Consul/Sentinel - 2 CPU, 2GiB memory per node +- 4 Sidekiq - 4 CPU, 16GiB memory per node +- 5 GitLab application nodes - 16 CPU, 64GiB memory per node +- 1 Gitaly - 16 CPU, 64GiB memory +- 1 Monitoring node - 2 CPU, 8GiB memory, 100GiB local storage ### Fully Distributed @@ -211,4 +214,3 @@ separately: 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) 1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) - diff --git a/doc/administration/high_availability/alpha_database.md b/doc/administration/high_availability/alpha_database.md index 7bf20be60e6..7afd739f44c 100644 --- a/doc/administration/high_availability/alpha_database.md +++ b/doc/administration/high_availability/alpha_database.md @@ -2,5 +2,4 @@ redirect_to: 'database.md' --- -This documentation has been moved to the main -[database documentation](database.md#configure_using_omnibus_for_high_availability). +This document was moved to [another location](database.md). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md index 49199b659bc..b02a61b9256 100644 --- a/doc/administration/high_availability/consul.md +++ b/doc/administration/high_availability/consul.md @@ -1,6 +1,8 @@ -# Working with the bundled Consul service **(PREMIUM ONLY)** +--- +type: reference +--- -## Overview +# Working with the bundled Consul service **(PREMIUM ONLY)** As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) that can be managed through `/etc/gitlab/gitlab.rb`. @@ -26,27 +28,27 @@ On each Consul node perform the following: 1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - ```ruby - # Disable all components except Consul - roles ['consul_role'] + ```ruby + # Disable all components except Consul + roles ['consul_role'] - # START user configuration - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - server: true, - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } + # START user configuration + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + server: true, + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } - # Disable auto migrations - gitlab_rails['auto_migrate'] = false - # - # END user configuration - ``` + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + # + # END user configuration + ``` - > `consul_role` was introduced with GitLab 10.3 + > `consul_role` was introduced with GitLab 10.3 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -77,6 +79,7 @@ check the [Troubleshooting section](#troubleshooting) before proceeding. ### Checking cluster membership To see which nodes are part of the cluster, run the following on any member in the cluster + ``` # /opt/gitlab/embedded/bin/consul members Node Address Status Type Build Protocol DC @@ -112,18 +115,18 @@ You will see messages like the following in `gitlab-ctl tail consul` output if y 2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader ``` - To fix this: 1. Pick an address on each node that all of the other nodes can reach this node through. 1. Update your `/etc/gitlab/gitlab.rb` - ```ruby - consul['configuration'] = { - ... - bind_addr: 'IP ADDRESS' - } - ``` + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` + 1. Run `gitlab-ctl reconfigure` If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node. @@ -144,12 +147,13 @@ To fix this: 1. Pick an address on the node that all of the other nodes can reach this node through. 1. Update your `/etc/gitlab/gitlab.rb` - ```ruby - consul['configuration'] = { - ... - bind_addr: 'IP ADDRESS' - } - ``` + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` + 1. Run `gitlab-ctl reconfigure` ### Outage recovery @@ -157,6 +161,7 @@ To fix this: If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion. #### Recreate from scratch + By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize ``` @@ -168,4 +173,5 @@ By default, GitLab does not store anything in the consul cluster that cannot be After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well. #### Recover a failed cluster + If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index c32a73080ff..7c9e02d889e 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -1,5 +1,12 @@ +--- +type: reference +--- + # Configuring PostgreSQL for Scaling and High Availability +In this section, you'll be guided through configuring a PostgreSQL database +to be used with GitLab in a highly available environment. + ## Provide your own PostgreSQL instance **(CORE ONLY)** If you're hosting GitLab on a cloud provider, you can optionally use a @@ -33,15 +40,15 @@ deploy the bundled PostgreSQL. 1. SSH into the PostgreSQL server. 1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - - Do not complete any other steps on the download page. + - Do not complete any other steps on the download page. 1. Generate a password hash for PostgreSQL. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next step as the value of `POSTGRESQL_PASSWORD_HASH`. - ```sh - sudo gitlab-ctl pg-password-md5 gitlab - ``` + ```sh + sudo gitlab-ctl pg-password-md5 gitlab + ``` 1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder values appropriately. @@ -51,32 +58,32 @@ deploy the bundled PostgreSQL. addresses of the GitLab application servers that will connect to the database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` - ```ruby - # Disable all components except PostgreSQL - roles ['postgres_role'] - repmgr['enable'] = false - consul['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - redis_exporter['enable'] = false - gitlab_monitor['enable'] = false - - postgresql['listen_address'] = '0.0.0.0' - postgresql['port'] = 5432 - - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - # ???? - postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - ``` + ```ruby + # Disable all components except PostgreSQL + roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + # ???? + postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` - NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 1. [Reconfigure GitLab] for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and @@ -194,9 +201,9 @@ When using default setup, minimum configuration requires: - `CONSUL_PASSWORD_HASH`. This is a hash generated out of consul username/password pair. Can be generated with: - ```sh - sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME - ``` + ```sh + sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME + ``` - `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. @@ -237,9 +244,9 @@ We will need the following password information for the application's database u - `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. Can be generated with: - ```sh - sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME - ``` + ```sh + sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME + ``` ##### Pgbouncer information @@ -250,9 +257,9 @@ When using default setup, minimum configuration requires: - `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of pgbouncer username/password pair. Can be generated with: - ```sh - sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME - ``` + ```sh + sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME + ``` - `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running Pgbouncer. @@ -288,7 +295,6 @@ Make sure you install the necessary dependencies from step 1, add GitLab package repository from step 2. When installing the GitLab package, do not supply `EXTERNAL_URL` value. - #### Configuring the Database nodes 1. Make sure to [configure the Consul nodes](consul.md). @@ -296,53 +302,55 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. 1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the consul agent - consul['services'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - # Replace X with value of number of db nodes + 1 - postgresql['max_wal_senders'] = X - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) - - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `postgres_role` was introduced with GitLab 10.3 + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + postgresql['max_replication_slots'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + > `postgres_role` was introduced with GitLab 10.3 1. On secondary nodes, add all the configuration specified above for primary node to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration to inform gitlab-ctl that they are standby nodes initially and it need not attempt to register them as primary node + ``` # HA setting to specify if a node should attempt to be master on initialization repmgr['master_on_initialization'] = false @@ -367,31 +375,31 @@ Select one node as a primary node. 1. Open a database prompt: - ```sh - gitlab-psql -d gitlabhq_production - ``` + ```sh + gitlab-psql -d gitlabhq_production + ``` 1. Enable the `pg_trgm` extension: - ```sh - CREATE EXTENSION pg_trgm; - ``` + ```sh + CREATE EXTENSION pg_trgm; + ``` 1. Exit the database prompt by typing `\q` and Enter. 1. Verify the cluster is initialized with one node: - ```sh - gitlab-ctl repmgr cluster show - ``` + ```sh + gitlab-ctl repmgr cluster show + ``` - The output should be similar to the following: + The output should be similar to the following: - ``` - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` + ``` + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` 1. Note down the hostname/ip in the connection string: `host=HOSTNAME`. We will refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value @@ -402,43 +410,43 @@ Select one node as a primary node. 1. Set up the repmgr standby: - ```sh - gitlab-ctl repmgr standby setup MASTER_NODE_NAME - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - # gitlab-ctl repmgr standby setup MASTER_NODE_NAME - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` + ```sh + gitlab-ctl repmgr standby setup MASTER_NODE_NAME + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` 1. Verify the node now appears in the cluster: - ```sh - gitlab-ctl repmgr cluster show - ``` + ```sh + gitlab-ctl repmgr cluster show + ``` - The output should be similar to the following: + The output should be similar to the following: - ``` - Role | Name | Upstream | Connection String - ----------+---------|-----------|------------------------------------------------ - * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` + ``` + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` Repeat the above steps on all secondary nodes. @@ -478,88 +486,7 @@ Check the [Troubleshooting section](#troubleshooting) before proceeding. #### Configuring the Pgbouncer node -1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. - -1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - - ```ruby - # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] - - # Configure Pgbouncer - pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - - # Configure Consul agent - consul['watchers'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # Replace CONSUL_PASSWORD_HASH with with a generated md5 value - # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value - pgbouncer['users'] = { - 'gitlab-consul': { - password: 'CONSUL_PASSWORD_HASH' - }, - 'pgbouncer': { - password: 'PGBOUNCER_PASSWORD_HASH' - } - } - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `pgbouncer_role` was introduced with GitLab 10.3 - -1. [Reconfigure GitLab] for the changes to take effect. - -1. Create a `.pgpass` file so Consul is able to - reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: - - ```sh - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` - -##### PGBouncer Checkpoint - -1. Ensure the node is talking to the current master: - - ```sh - gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD - ``` - - If there is an error `psql: ERROR: Auth failed` after typing in the - password, ensure you previously generated the MD5 password hashes with the correct - format. The correct format is to concatenate the password and the username: - `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text - needed to generate an MD5 password hash for the `pgbouncer` user. - -1. Once the console prompt is available, run the following queries: - - ```sh - show databases ; show clients ; - ``` - - The output should be similar to the following: - - ``` - name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections - ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- - gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 - pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 - (2 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls - ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- - C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | - (2 rows) - ``` +See our [documentation for Pgbouncer](pgbouncer.md) for information on running Pgbouncer as part of an HA setup. #### Configuring the Application nodes @@ -568,15 +495,15 @@ attributes set, but the following need to be set. 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Disable PostgreSQL on the application node - postgresql['enable'] = false + ```ruby + # Disable PostgreSQL on the application node + postgresql['enable'] = false - gitlab_rails['db_host'] = 'PGBOUNCER_NODE' - gitlab_rails['db_port'] = 6432 - gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' - gitlab_rails['auto_migrate'] = false - ``` + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' + gitlab_rails['auto_migrate'] = false + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -736,45 +663,45 @@ After deploying the configuration follow these steps: 1. On `10.6.0.21`, our primary database - Enable the `pg_trgm` extension + Enable the `pg_trgm` extension - ```sh - gitlab-psql -d gitlabhq_production - ``` + ```sh + gitlab-psql -d gitlabhq_production + ``` - ``` - CREATE EXTENSION pg_trgm; - ``` + ``` + CREATE EXTENSION pg_trgm; + ``` 1. On `10.6.0.22`, our first standby database - Make this node a standby of the primary + Make this node a standby of the primary - ```sh - gitlab-ctl repmgr standby setup 10.6.0.21 - ``` + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` 1. On `10.6.0.23`, our second standby database - Make this node a standby of the primary + Make this node a standby of the primary - ```sh - gitlab-ctl repmgr standby setup 10.6.0.21 - ``` + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` 1. On `10.6.0.31`, our application server - Set gitlab-consul's pgbouncer password to `toomanysecrets` + Set gitlab-consul's pgbouncer password to `toomanysecrets` - ```sh - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` - Run database migrations + Run database migrations - ```sh - gitlab-rake gitlab:db:configure - ``` + ```sh + gitlab-rake gitlab:db:configure + ``` #### Example minimal setup @@ -911,16 +838,16 @@ standby nodes. 1. Ensure the old master node is not still active. 1. Login to the server that should become the new master and run: - ```sh - gitlab-ctl repmgr standby promote - ``` + ```sh + gitlab-ctl repmgr standby promote + ``` 1. If there are any other standby servers in the cluster, have them follow the new master server: - ```sh - gitlab-ctl repmgr standby follow NEW_MASTER - ``` + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + ``` #### Restore procedure @@ -930,42 +857,42 @@ after it has been restored to service. - If you want to remove the node from the cluster, on any other node in the cluster, run: - ```sh - gitlab-ctl repmgr standby unregister --node=X - ``` + ```sh + gitlab-ctl repmgr standby unregister --node=X + ``` - where X is the value of node in `repmgr.conf` on the old server. + where X is the value of node in `repmgr.conf` on the old server. - To find this, you can use: + To find this, you can use: - ```sh - awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf - ``` + ```sh + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` - It will output something like: + It will output something like: - ``` - 959789412 - ``` + ``` + 959789412 + ``` - Then you will use this id to unregister the node: + Then you will use this id to unregister the node: - ```sh - gitlab-ctl repmgr standby unregister --node=959789412 - ``` + ```sh + gitlab-ctl repmgr standby unregister --node=959789412 + ``` - To add the node as a standby server: - ```sh - gitlab-ctl repmgr standby follow NEW_MASTER - gitlab-ctl restart repmgrd - ``` + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` - CAUTION: **Warning:** When the server is brought back online, and before - you switch it to a standby node, repmgr will report that there are two masters. - If there are any clients that are still attempting to write to the old master, - this will cause a split, and the old master will need to be resynced from - scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + CAUTION: **Warning:** When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. #### Alternate configurations @@ -1008,13 +935,13 @@ the previous section: 1. On the current master node, create a password for the `gitlab` and `gitlab_repmgr` user: - ```sh - gitlab-psql -d template1 - template1=# \password gitlab_repmgr - Enter password: **** - Confirm password: **** - template1=# \password gitlab - ``` + ```sh + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` 1. On each database node: @@ -1028,9 +955,9 @@ the previous section: 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to when asked: - ```sh - gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' - ``` + ```sh + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` 1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`: 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for @@ -1058,7 +985,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. ## Troubleshooting -### Consul and PostgreSQL changes not taking effect. +### Consul and PostgreSQL changes not taking effect Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index b7eaa4ce105..739d1ae35fb 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring Gitaly for Scaled and High Availability Gitaly does not yet support full high availability. However, Gitaly is quite @@ -46,3 +50,15 @@ Continue configuration of other components by going back to: ``` 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 9b1b7142e83..8818a9606de 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -1,4 +1,8 @@ -# Configuring GitLab Scaling and High Availability +--- +type: reference +--- + +# Configuring GitLab for Scaling and High Availability > **Note:** There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand @@ -7,33 +11,33 @@ 1. If necessary, install the NFS client utility packages using the following commands: - ``` - # Ubuntu/Debian - apt-get install nfs-common + ``` + # Ubuntu/Debian + apt-get install nfs-common - # CentOS/Red Hat - yum install nfs-utils nfs-utils-lib - ``` + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` 1. Specify the necessary NFS shares. Mounts are specified in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose to configure your NFS server. See [NFS documentation](nfs.md) for the various options. Here is an example snippet to add to `/etc/fstab`: - ``` - 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - ``` + ``` + 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 + ``` 1. Create the shared directories. These may be different depending on your NFS mount locations. - ``` - mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data - ``` + ``` + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` 1. Download/install GitLab Omnibus using **steps 1 and 2** from [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other @@ -46,52 +50,52 @@ added NFS mounts in the default data locations. Additionally the UID and GIDs given are just examples and you should configure with your preferred values. - ```ruby - external_url 'https://gitlab.example.com' - - # Prevent GitLab from starting if NFS data mounts are not available - high_availability['mountpoint'] = '/var/opt/gitlab/git-data' - - # Disable components that will not be on the GitLab application server - roles ['application_role'] - nginx['enable'] = true - - # PostgreSQL connection details - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server - gitlab_rails['db_password'] = 'DB password' - - # Redis connection details - gitlab_rails['redis_port'] = '6379' - gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server - gitlab_rails['redis_password'] = 'Redis Password' - - # Ensure UIDs and GIDs match between servers for permissions via NFS - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` + ```ruby + external_url 'https://gitlab.example.com' + + # Prevent GitLab from starting if NFS data mounts are not available + high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + + # Disable components that will not be on the GitLab application server + roles ['application_role'] + nginx['enable'] = true + + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + + # Redis connection details + gitlab_rails['redis_port'] = '6379' + gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server + gitlab_rails['redis_password'] = 'Redis Password' + + # Ensure UIDs and GIDs match between servers for permissions via NFS + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` 1. [Enable monitoring](#enable-monitoring) - > **Note:** To maintain uniformity of links across HA clusters, the `external_url` - on the first application server as well as the additional application - servers should point to the external url that users will use to access GitLab. - In a typical HA setup, this will be the url of the load balancer which will - route traffic to all GitLab application servers in the HA cluster. - > - > **Note:** When you specify `https` in the `external_url`, as in the example - above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If - certificates are not present, Nginx will fail to start. See - [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) - for more information. - > - > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure - of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. + > **Note:** To maintain uniformity of links across HA clusters, the `external_url` + on the first application server as well as the additional application + servers should point to the external url that users will use to access GitLab. + In a typical HA setup, this will be the url of the load balancer which will + route traffic to all GitLab application servers in the HA cluster. + > + > **Note:** When you specify `https` in the `external_url`, as in the example + above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If + certificates are not present, Nginx will fail to start. See + [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) + for more information. + > + > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure + of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. ## First GitLab application server @@ -114,12 +118,12 @@ need some extra configuration. secondary servers **prior to** running the first `reconfigure` in the steps above. - ```ruby - gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' - gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' - gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' - gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' - ``` + ```ruby + gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' + gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' + gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' + gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' + ``` 1. Run `touch /etc/gitlab/skip-auto-reconfigure` to prevent database migrations from running on upgrade. Only the primary GitLab application server should diff --git a/doc/administration/high_availability/img/fully-distributed.png b/doc/administration/high_availability/img/fully-distributed.png Binary files differindex ad23207134e..c3cd2bf24f0 100644 --- a/doc/administration/high_availability/img/fully-distributed.png +++ b/doc/administration/high_availability/img/fully-distributed.png diff --git a/doc/administration/high_availability/img/horizontal.png b/doc/administration/high_availability/img/horizontal.png Binary files differindex c3bd489d96f..75d08e1097a 100644 --- a/doc/administration/high_availability/img/horizontal.png +++ b/doc/administration/high_availability/img/horizontal.png diff --git a/doc/administration/high_availability/img/hybrid.png b/doc/administration/high_availability/img/hybrid.png Binary files differindex 7d4a56bf0ea..8dd9923e597 100644 --- a/doc/administration/high_availability/img/hybrid.png +++ b/doc/administration/high_availability/img/hybrid.png diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index 28b226cacd5..f11d27487d1 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Load Balancer for GitLab HA In an active/active GitLab configuration, you will need a load balancer to route @@ -56,11 +60,21 @@ for details on managing SSL certificates and configuring Nginx. ### Basic ports -| LB Port | Backend Port | Protocol | -| ------- | ------------ | --------------- | -| 80 | 80 | HTTP [^1] | -| 443 | 443 | TCP or HTTPS [^1] [^2] | -| 22 | 22 | TCP | +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. ### GitLab Pages Ports @@ -68,12 +82,19 @@ If you're using GitLab Pages with custom domain support you will need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the -[GitLab Pages documentation][gitlab-pages] for more information. +[GitLab Pages documentation](../pages/index.md) for more information. -| LB Port | Backend Port | Protocol | -| ------- | ------------ | -------- | -| 80 | Varies [^3] | HTTP | -| 443 | Varies [^3] | TCP [^4] | +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. ### Alternate SSH Port @@ -82,7 +103,7 @@ it may be helpful to configure an alternate SSH hostname that allows users to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address compared to the other GitLab HTTP configuration above. -Configure DNS for an alternate SSH hostname such as altssh.gitlab.example.com. +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | LB Port | Backend Port | Protocol | | ------- | ------------ | -------- | @@ -97,20 +118,14 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) -[^1]: [Web terminal](../../ci/environments.md#web-terminals) support requires - your load balancer to correctly handle WebSocket connections. When using - HTTP or HTTPS proxying, this means your load balancer must be configured - to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the - [web terminal](../integration/terminal.md) integration guide for - more details. -[^2]: When using HTTPS protocol for port 443, you will need to add an SSL - certificate to the load balancers. If you wish to terminate SSL at the - GitLab application server instead, use TCP protocol. -[^3]: The backend port for GitLab Pages depends on the - `gitlab_pages['external_http']` and `gitlab_pages['external_https']` - setting. See [GitLab Pages documentation][gitlab-pages] for more details. -[^4]: Port 443 for GitLab Pages should always use the TCP protocol. Users can - configure custom domains with custom SSL, which would not be possible - if SSL was terminated at the load balancer. - -[gitlab-pages]: ../pages/index.md +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index 385e7441ac9..b2750603c74 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -1,10 +1,16 @@ +--- +type: reference +--- + # Configuring a Monitoring node for Scaling and High Availability > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. +You can configure a Prometheus node to monitor GitLab. + ## Standalone Monitoring node using GitLab Omnibus -The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana. +The GitLab Omnibus package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md). The monitoring node is not highly available. See [Scaling and High Availability](README.md) for an overview of GitLab scaling and high availability options. @@ -12,7 +18,7 @@ The steps below are the minimum necessary to configure a Monitoring node running Omnibus: 1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install) the Omnibus GitLab +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. @@ -20,44 +26,44 @@ Omnibus: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - external_url 'http://gitlab.example.com' - - # Enable Prometheus - prometheus['enable'] = true - prometheus['listen_address'] = '0.0.0.0:9090' - prometheus['monitor_kubernetes'] = false - - # Enable Grafana - grafana['enable'] = true - grafana['admin_password'] = 'toomanysecrets' - - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } - - # Disable all other services - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_monitor['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - ``` + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + ``` 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. @@ -67,3 +73,15 @@ Once monitoring using Service Discovery is enabled with `consul['monitoring_serv ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both `consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` will result in errors. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 561ba214686..274bd32299b 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # NFS You can view information and options set for each of the mounted NFS file @@ -42,34 +46,20 @@ maintaining ID mapping without LDAP, in most cases you should enable numeric UID and GIDs (which is off by default in some cases) for simplified permission management between systems: - - [NetApp instructions](https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html) - - For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping) +- [NetApp instructions](https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html) +- For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping) ### Improving NFS performance with GitLab -NOTE: **Note:** This is only available starting in certain versions of GitLab: 11.5.11, -11.6.11, 11.7.12, 11.8.8, 11.9.0 and up (e.g. 11.10, 11.11, etc.) +NOTE: **Note:** From GitLab 12.1, it will automatically be detected if Rugged can and should be used per storage. -If you are using NFS to share Git data, we recommend that you enable a -number of feature flags that will allow GitLab application processes to -access Git data directly instead of going through the [Gitaly -service](../gitaly/index.md). Depending on your workload and disk -performance, these flags may help improve performance. See [the -issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/57317) for more -details. - -To do this, run the Rake task: +If you previously enabled Rugged using the feature flag, you will need to unset the feature flag by using: ```sh -sudo gitlab-rake gitlab:features:enable_rugged +sudo gitlab-rake gitlab:features:unset_rugged ``` -If you need to undo this setting for some reason such as switching to [Gitaly without NFS](gitaly.md) -(recommended), run: - -```sh -sudo gitlab-rake gitlab:features:disable_rugged -``` +If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. ### Known issues @@ -81,16 +71,21 @@ bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) that may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). +NOTE: **Note** Red Hat Enterprise 7 [shipped a kernel +update](https://access.redhat.com/errata/RHSA-2019:2029) on August 6, +2019 that may have resolved this problem. The following instructions may +not be needed if the latest kernel is updated properly. + GitLab recommends all NFS users disable the NFS server delegation feature. To disable NFS server delegations on an Linux NFS server, do the following: 1. On the NFS server, run: - ```sh - echo 0 > /proc/sys/fs/leases-enable - sysctl -w fs.leases-enable=0 - ``` + ```sh + echo 0 > /proc/sys/fs/leases-enable + sysctl -w fs.leases-enable=0 + ``` 1. Restart the NFS server process. For example, on CentOS run `service nfs restart`. @@ -236,3 +231,15 @@ Read more on high-availability configuration: 1. [Configure the load balancers](load_balancer.md) [udp-log-shipping]: https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md index a8d69b9ab0a..9b0e085fe25 100644 --- a/doc/administration/high_availability/nfs_host_client_setup.md +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring NFS for GitLab HA Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster @@ -133,3 +137,15 @@ client with the command below. ```sh sudo ufw allow from <client-ip-address> to any port nfs ``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 053dae25823..b99724d12a2 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -1,6 +1,8 @@ -# Working with the bundle Pgbouncer service +--- +type: reference +--- -## Overview +# Working with the bundle Pgbouncer service As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. @@ -14,7 +16,89 @@ It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on i ### Running Pgbouncer as part of an HA GitLab installation -See our [HA documentation for PostgreSQL](database.md) for information on running pgbouncer as part of a HA setup +1. Make sure you collect [`CONSUL_SERVER_NODES`](database.md#consul-information), [`CONSUL_PASSWORD_HASH`](database.md#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](database.md#pgbouncer-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure Pgbouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + NOTE: **Note:** + `pgbouncer_role` was introduced with GitLab 10.3. + +1. Run `gitlab-ctl reconfigure` + +1. Create a `.pgpass` file so Consul is able to + reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +#### PGBouncer Checkpoint + +1. Ensure the node is talking to the current master: + + ```sh + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. + +1. Once the console prompt is available, run the following queries: + + ```sh + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ``` + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` ### Running Pgbouncer as part of a non-HA GitLab installation @@ -24,39 +108,39 @@ See our [HA documentation for PostgreSQL](database.md) for information on runnin 1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` - ```ruby - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' - postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' - postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on - postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node - ``` + ```ruby + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' + postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' + postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on + postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node + ``` 1. Run `gitlab-ctl reconfigure` - **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. + **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. 1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` - ```ruby - pgbouncer['enable'] = true - pgbouncer['databases'] = { - gitlabhq_production: { - host: 'DATABASE_HOST', - user: 'pgbouncer', - password: 'PGBOUNCER_USER_PASSWORD_HASH' - } - } - ``` + ```ruby + pgbouncer['enable'] = true + pgbouncer['databases'] = { + gitlabhq_production: { + host: 'DATABASE_HOST', + user: 'pgbouncer', + password: 'PGBOUNCER_USER_PASSWORD_HASH' + } + } + ``` 1. Run `gitlab-ctl reconfigure` 1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb` - ```ruby - gitlab_rails['db_host'] = 'PGBOUNCER_HOST' - gitlab_rails['db_port'] = '6432' - gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' - ``` + ```ruby + gitlab_rails['db_host'] = 'PGBOUNCER_HOST' + gitlab_rails['db_port'] = '6432' + gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' + ``` 1. Run `gitlab-ctl reconfigure` @@ -66,28 +150,28 @@ See our [HA documentation for PostgreSQL](database.md) for information on runnin > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. - If you enable Monitoring, it must be enabled on **all** pgbouncer servers. +If you enable Monitoring, it must be enabled on **all** pgbouncer servers. - 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' - ``` + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' + ``` - 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. ### Interacting with pgbouncer @@ -109,6 +193,7 @@ pgbouncer=# The password you will be prompted for is the PGBOUNCER_USER_PASSWORD To get some basic information about the instance, run + ```shell pgbouncer=# show databases; show clients; show servers; name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 27310c59755..1b79dde9476 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring Redis for Scaling and High Availability ## Provide your own Redis instance **(CORE ONLY)** @@ -47,28 +51,28 @@ Omnibus: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - ## Enable Redis - redis['enable'] = true - - ## Disable all other services - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - unicorn['enable'] = false - postgresql['enable'] = false - nginx['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - gitlab_monitor['enable'] = false - gitaly['enable'] = false - - redis['bind'] = '0.0.0.0' - redis['port'] = '6379' - redis['password'] = 'SECRET_PASSWORD_HERE' - - gitlab_rails['auto_migrate'] = false - ``` + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_monitor['enable'] = false + gitaly['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = '6379' + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['auto_migrate'] = false + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and @@ -359,37 +363,37 @@ The prerequisites for a HA Redis setup are the following: 1. SSH into the **master** Redis server. 1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. - # If you really need to bind to an external accessible IP, make - # sure you add extra firewall rules to prevent unauthorized access. - redis['bind'] = '10.0.0.1' + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.1' - # Define a port so Redis can listen for TCP requests which will allow other - # machines to connect to it. - redis['port'] = 6379 + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'redis-password-goes-here' - ``` + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'redis-password-goes-here' + ``` 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following configuration to your `/etc/gitlab/gitlab.rb` file: - ``` - gitlab_rails['auto_migrate'] = false - ``` + ``` + gitlab_rails['auto_migrate'] = false + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. @@ -402,42 +406,42 @@ The prerequisites for a HA Redis setup are the following: 1. SSH into the **slave** Redis server. 1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - ```ruby - # Specify server role as 'redis_slave_role' - roles ['redis_slave_role'] + ```ruby + # Specify server role as 'redis_slave_role' + roles ['redis_slave_role'] - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. - # If you really need to bind to an external accessible IP, make - # sure you add extra firewall rules to prevent unauthorized access. - redis['bind'] = '10.0.0.2' + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.0.0.2' - # Define a port so Redis can listen for TCP requests which will allow other - # machines to connect to it. - redis['port'] = 6379 + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 - # The same password for Redis authentication you set up for the master node. - redis['password'] = 'redis-password-goes-here' + # The same password for Redis authentication you set up for the master node. + redis['password'] = 'redis-password-goes-here' - # The IP of the master Redis node. - redis['master_ip'] = '10.0.0.1' + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' - # Port of master Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ``` + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + ``` 1. To prevent reconfigure from running automatically on upgrade, run: - ``` - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` + ``` + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. @@ -487,89 +491,89 @@ multiple machines with the Sentinel daemon. 1. **You can omit this step if the Sentinels will be hosted in the same node as the other Redis instances.** - [Download/install](https://about.gitlab.com/downloads-ee) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. + [Download/install](https://about.gitlab.com/downloads-ee) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the Sentinels in the same node as the other Redis instances, some values might be duplicate below): - ```ruby - roles ['redis_sentinel_role'] - - # Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' - - # The same password for Redis authentication you set up for the master node. - redis['master_password'] = 'redis-password-goes-here' - - # The IP of the master Redis node. - redis['master_ip'] = '10.0.0.1' - - # Define a port so Redis can listen for TCP requests which will allow other - # machines to connect to it. - redis['port'] = 6379 - - # Port of master Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel - sentinel['bind'] = '10.0.0.1' - - # Port that Sentinel listens on, uncomment to change to non default. Defaults - # to `26379`. - # sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to master failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the master. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the master being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - # sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same master by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a slave replicating to a wrong master according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right master, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (SLAVEOF NO ONE yet not - ## acknowledged by the promoted slave). - ## - ## - The maximum time a failover in progress waits for all the slaves to be - ## reconfigured as slaves of the new master. However even after this time - ## the slaves will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - # sentinel['failover_timeout'] = 60000 - ``` + ```ruby + roles ['redis_sentinel_role'] + + # Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + # The same password for Redis authentication you set up for the master node. + redis['master_password'] = 'redis-password-goes-here' + + # The IP of the master Redis node. + redis['master_ip'] = '10.0.0.1' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Port of master Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel + sentinel['bind'] = '10.0.0.1' + + # Port that Sentinel listens on, uncomment to change to non default. Defaults + # to `26379`. + # sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to master failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the master. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the master being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + # sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## - The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + # sentinel['failover_timeout'] = 60000 + ``` 1. To prevent database migrations from running on upgrade, run: - ``` - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` + ``` + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` - Only the primary GitLab application server should handle migrations. + Only the primary GitLab application server should handle migrations. 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes. @@ -593,20 +597,20 @@ which ideally should not have Redis or Sentinels on it for a HA setup. 1. SSH into the server where the GitLab application is installed. 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: - ``` - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' - - ## The same password for Redis authentication you set up for the master node. - redis['master_password'] = 'redis-password-goes-here' - - ## A list of sentinels with `host` and `port` - gitlab_rails['redis_sentinels'] = [ - {'host' => '10.0.0.1', 'port' => 26379}, - {'host' => '10.0.0.2', 'port' => 26379}, - {'host' => '10.0.0.3', 'port' => 26379} - ] - ``` + ```ruby + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + ## The same password for Redis authentication you set up for the master node. + redis['master_password'] = 'redis-password-goes-here' + + ## A list of sentinels with `host` and `port` + gitlab_rails['redis_sentinels'] = [ + {'host' => '10.0.0.1', 'port' => 26379}, + {'host' => '10.0.0.2', 'port' => 26379}, + {'host' => '10.0.0.3', 'port' => 26379} + ] + ``` 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. @@ -791,31 +795,34 @@ cache, queues, and shared_state. To make this work with Sentinel: 1. Set the appropriate variable in `/etc/gitlab/gitlab.rb` for each instance you are using: - ```ruby - gitlab_rails['redis_cache_instance'] = REDIS_CACHE_URL - gitlab_rails['redis_queues_instance'] = REDIS_QUEUES_URL - gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL - ``` + ```ruby + gitlab_rails['redis_cache_instance'] = REDIS_CACHE_URL + gitlab_rails['redis_queues_instance'] = REDIS_QUEUES_URL + gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL + ``` + **Note**: Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_MASTER_NAME` - 1. PASSWORD is the plaintext password for the Redis instance - 1. SENTINEL_MASTER_NAME is the Sentinel master name (e.g. `gitlab-redis-cache`) + 1. PASSWORD is the plaintext password for the Redis instance + 1. SENTINEL_MASTER_NAME is the Sentinel master name (e.g. `gitlab-redis-cache`) + 1. Include an array of hashes with host/port combinations, such as the following: - ```ruby - gitlab_rails['redis_cache_sentinels'] = [ - { host: REDIS_CACHE_SENTINEL_HOST, port: PORT1 }, - { host: REDIS_CACHE_SENTINEL_HOST2, port: PORT2 } - ] - gitlab_rails['redis_queues_sentinels'] = [ - { host: REDIS_QUEUES_SENTINEL_HOST, port: PORT1 }, - { host: REDIS_QUEUES_SENTINEL_HOST2, port: PORT2 } - ] - gitlab_rails['redis_shared_state_sentinels'] = [ - { host: SHARED_STATE_SENTINEL_HOST, port: PORT1 }, - { host: SHARED_STATE_SENTINEL_HOST2, port: PORT2 } - ] - ``` + ```ruby + gitlab_rails['redis_cache_sentinels'] = [ + { host: REDIS_CACHE_SENTINEL_HOST, port: PORT1 }, + { host: REDIS_CACHE_SENTINEL_HOST2, port: PORT2 } + ] + gitlab_rails['redis_queues_sentinels'] = [ + { host: REDIS_QUEUES_SENTINEL_HOST, port: PORT1 }, + { host: REDIS_QUEUES_SENTINEL_HOST2, port: PORT2 } + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + { host: SHARED_STATE_SENTINEL_HOST, port: PORT1 }, + { host: SHARED_STATE_SENTINEL_HOST2, port: PORT2 } + ] + ``` + 1. Note that for each persistence class, GitLab will default to using the configuration specified in `gitlab_rails['redis_sentinels']` unless overridden by the settings above. @@ -879,12 +886,12 @@ in order for the HA setup to work as expected. Before proceeding with the troubleshooting below, check your firewall rules: - Redis machines - - Accept TCP connection in `6379` - - Connect to the other Redis machines via TCP in `6379` + - Accept TCP connection in `6379` + - Connect to the other Redis machines via TCP in `6379` - Sentinel machines - - Accept TCP connection in `26379` - - Connect to other Sentinel machines via TCP in `26379` - - Connect to the Redis machines via TCP in `6379` + - Accept TCP connection in `26379` + - Connect to other Sentinel machines via TCP in `26379` + - Connect to the Redis machines via TCP in `6379` ### Troubleshooting Redis replication @@ -952,38 +959,38 @@ To make sure your configuration is correct: 1. SSH into your GitLab application server 1. Enter the Rails console: - ``` - # For Omnibus installations - sudo gitlab-rails console + ``` + # For Omnibus installations + sudo gitlab-rails console - # For source installations - sudo -u git rails console production - ``` + # For source installations + sudo -u git rails console production + ``` 1. Run in the console: - ```ruby - redis = Redis.new(Gitlab::Redis::SharedState.params) - redis.info - ``` + ```ruby + redis = Redis.new(Gitlab::Redis::SharedState.params) + redis.info + ``` - Keep this screen open and try to simulate a failover below. + Keep this screen open and try to simulate a failover below. 1. To simulate a failover on master Redis, SSH into the Redis server and run: - ```bash - # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one - redis-cli -h localhost -p 6379 DEBUG sleep 20 - ``` + ```bash + # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one + redis-cli -h localhost -p 6379 DEBUG sleep 20 + ``` 1. Then back in the Rails console from the first step, run: - ``` - redis.info - ``` + ``` + redis.info + ``` - You should see a different port after a few seconds delay - (the failover/reconnect time). + You should see a different port after a few seconds delay + (the failover/reconnect time). ## Changelog diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index be6b547372a..63915e5d96c 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring non-Omnibus Redis for GitLab HA This is the documentation for configuring a Highly Available Redis setup when @@ -49,22 +53,22 @@ Assuming that the Redis master instance IP is `10.0.0.1`: 1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: - ```conf - ## Define a `bind` address pointing to a local IP that your other machines - ## can reach you. If you really need to bind to an external accessible IP, make - ## sure you add extra firewall rules to prevent unauthorized access: - bind 10.0.0.1 + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 - ## Define a `port` to force redis to listen on TCP so other machines can - ## connect to it (default port is `6379`). - port 6379 + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 - ## Set up password authentication (use the same password in all nodes). - ## The password should be defined equal for both `requirepass` and `masterauth` - ## when setting up Redis to use with Sentinel. - requirepass redis-password-goes-here - masterauth redis-password-goes-here - ``` + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` 1. Restart the Redis service for the changes to take effect. @@ -75,25 +79,25 @@ Assuming that the Redis slave instance IP is `10.0.0.2`: 1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: - ```conf - ## Define a `bind` address pointing to a local IP that your other machines - ## can reach you. If you really need to bind to an external accessible IP, make - ## sure you add extra firewall rules to prevent unauthorized access: - bind 10.0.0.2 + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.2 - ## Define a `port` to force redis to listen on TCP so other machines can - ## connect to it (default port is `6379`). - port 6379 + ## Define a `port` to force redis to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 6379 - ## Set up password authentication (use the same password in all nodes). - ## The password should be defined equal for both `requirepass` and `masterauth` - ## when setting up Redis to use with Sentinel. - requirepass redis-password-goes-here - masterauth redis-password-goes-here + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here - ## Define `slaveof` pointing to the Redis master instance with IP and port. - slaveof 10.0.0.1 6379 - ``` + ## Define `slaveof` pointing to the Redis master instance with IP and port. + slaveof 10.0.0.1 6379 + ``` 1. Restart the Redis service for the changes to take effect. 1. Go through the steps again for all the other slave nodes. @@ -110,56 +114,57 @@ master with IP `10.0.0.1` (some settings might overlap with the master): 1. [Install Redis Sentinel](https://redis.io/topics/sentinel) 1. Edit `/etc/redis/sentinel.conf`: - ```conf - ## Define a `bind` address pointing to a local IP that your other machines - ## can reach you. If you really need to bind to an external accessible IP, make - ## sure you add extra firewall rules to prevent unauthorized access: - bind 10.0.0.1 - - ## Define a `port` to force Sentinel to listen on TCP so other machines can - ## connect to it (default port is `6379`). - port 26379 - - ## Set up password authentication (use the same password in all nodes). - ## The password should be defined equal for both `requirepass` and `masterauth` - ## when setting up Redis to use with Sentinel. - requirepass redis-password-goes-here - masterauth redis-password-goes-here - - ## Define with `sentinel auth-pass` the same shared password you have - ## defined for both Redis master and slaves instances. - sentinel auth-pass gitlab-redis redis-password-goes-here - - ## Define with `sentinel monitor` the IP and port of the Redis - ## master node, and the quorum required to start a failover. - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - - ## Define with `sentinel down-after-milliseconds` the time in `ms` - ## that an unresponsive server will be considered down. - sentinel down-after-milliseconds gitlab-redis 10000 - - ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple - ## meanings: - ## - ## * The time needed to re-start a failover after a previous failover was - ## already tried against the same master by a given Sentinel, is two - ## times the failover timeout. - ## - ## * The time needed for a slave replicating to a wrong master according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right master, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## * The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (SLAVEOF NO ONE yet not - ## acknowledged by the promoted slave). - ## - ## * The maximum time a failover in progress waits for all the slaves to be - ## reconfigured as slaves of the new master. However even after this time - ## the slaves will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - sentinel failover_timeout 30000 - ``` + ```conf + ## Define a `bind` address pointing to a local IP that your other machines + ## can reach you. If you really need to bind to an external accessible IP, make + ## sure you add extra firewall rules to prevent unauthorized access: + bind 10.0.0.1 + + ## Define a `port` to force Sentinel to listen on TCP so other machines can + ## connect to it (default port is `6379`). + port 26379 + + ## Set up password authentication (use the same password in all nodes). + ## The password should be defined equal for both `requirepass` and `masterauth` + ## when setting up Redis to use with Sentinel. + requirepass redis-password-goes-here + masterauth redis-password-goes-here + + ## Define with `sentinel auth-pass` the same shared password you have + ## defined for both Redis master and slaves instances. + sentinel auth-pass gitlab-redis redis-password-goes-here + + ## Define with `sentinel monitor` the IP and port of the Redis + ## master node, and the quorum required to start a failover. + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + + ## Define with `sentinel down-after-milliseconds` the time in `ms` + ## that an unresponsive server will be considered down. + sentinel down-after-milliseconds gitlab-redis 10000 + + ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple + ## meanings: + ## + ## * The time needed to re-start a failover after a previous failover was + ## already tried against the same master by a given Sentinel, is two + ## times the failover timeout. + ## + ## * The time needed for a slave replicating to a wrong master according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right master, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## * The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (SLAVEOF NO ONE yet not + ## acknowledged by the promoted slave). + ## + ## * The maximum time a failover in progress waits for all the slaves to be + ## reconfigured as slaves of the new master. However even after this time + ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + sentinel failover_timeout 30000 + ``` + 1. Restart the Redis service for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes. @@ -180,21 +185,21 @@ setup: [resque.yml.example][resque], and uncomment the Sentinel lines, pointing to the correct server credentials: - ```yaml - # resque.yaml - production: - url: redis://:redi-password-goes-here@gitlab-redis/ - sentinels: - - - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port - ``` + ```yaml + # resque.yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` 1. [Restart GitLab][restart] for the changes to take effect. @@ -232,23 +237,23 @@ or a failover promotes a different **Master** node. 1. In `/etc/redis/redis.conf`: - ```conf - bind 10.0.0.1 - port 6379 - requirepass redis-password-goes-here - masterauth redis-password-goes-here - ``` + ```conf + bind 10.0.0.1 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + ``` 1. In `/etc/redis/sentinel.conf`: - ```conf - bind 10.0.0.1 - port 26379 - sentinel auth-pass gitlab-redis redis-password-goes-here - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - sentinel down-after-milliseconds gitlab-redis 10000 - sentinel failover_timeout 30000 - ``` + ```conf + bind 10.0.0.1 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` 1. Restart the Redis service for the changes to take effect. @@ -256,24 +261,24 @@ or a failover promotes a different **Master** node. 1. In `/etc/redis/redis.conf`: - ```conf - bind 10.0.0.2 - port 6379 - requirepass redis-password-goes-here - masterauth redis-password-goes-here - slaveof 10.0.0.1 6379 - ``` + ```conf + bind 10.0.0.2 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` 1. In `/etc/redis/sentinel.conf`: - ```conf - bind 10.0.0.2 - port 26379 - sentinel auth-pass gitlab-redis redis-password-goes-here - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - sentinel down-after-milliseconds gitlab-redis 10000 - sentinel failover_timeout 30000 - ``` + ```conf + bind 10.0.0.2 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` 1. Restart the Redis service for the changes to take effect. @@ -281,24 +286,24 @@ or a failover promotes a different **Master** node. 1. In `/etc/redis/redis.conf`: - ```conf - bind 10.0.0.3 - port 6379 - requirepass redis-password-goes-here - masterauth redis-password-goes-here - slaveof 10.0.0.1 6379 - ``` + ```conf + bind 10.0.0.3 + port 6379 + requirepass redis-password-goes-here + masterauth redis-password-goes-here + slaveof 10.0.0.1 6379 + ``` 1. In `/etc/redis/sentinel.conf`: - ```conf - bind 10.0.0.3 - port 26379 - sentinel auth-pass gitlab-redis redis-password-goes-here - sentinel monitor gitlab-redis 10.0.0.1 6379 2 - sentinel down-after-milliseconds gitlab-redis 10000 - sentinel failover_timeout 30000 - ``` + ```conf + bind 10.0.0.3 + port 26379 + sentinel auth-pass gitlab-redis redis-password-goes-here + sentinel monitor gitlab-redis 10.0.0.1 6379 2 + sentinel down-after-milliseconds gitlab-redis 10000 + sentinel failover_timeout 30000 + ``` 1. Restart the Redis service for the changes to take effect. @@ -306,20 +311,20 @@ or a failover promotes a different **Master** node. 1. Edit `/home/git/gitlab/config/resque.yml`: - ```yaml - production: - url: redis://:redi-password-goes-here@gitlab-redis/ - sentinels: - - - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port - - - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port - ``` + ```yaml + production: + url: redis://:redi-password-goes-here@gitlab-redis/ + sentinels: + - + host: 10.0.0.1 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.2 + port: 26379 # point to sentinel, not to redis port + - + host: 10.0.0.3 + port: 26379 # point to sentinel, not to redis port + ``` 1. [Restart GitLab][restart] for the changes to take effect. diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 1b01419e062..abe39d188aa 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -2,7 +2,6 @@ > [Introduced][ce-2371] in GitLab 8.4. ---- ## Automatic housekeeping GitLab automatically runs `git gc` and `git repack` on repositories @@ -32,8 +31,6 @@ the `pushes_since_gc` value is 200 a `git gc` will be run. You can find this option under your project's **Settings > General > Advanced**. ---- -  [ce-2371]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2371 "Housekeeping merge request" diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differindex 4f25c471908..e7d5c157d08 100644 --- a/doc/administration/img/custom_hooks_error_msg.png +++ b/doc/administration/img/custom_hooks_error_msg.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 73a39a6dd35..29915cb3a99 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -151,7 +151,7 @@ Reply by email should now be working. #### Postfix -Example configuration for Postfix mail server. Assumes mailbox incoming@gitlab.example.com. +Example configuration for Postfix mail server. Assumes mailbox `incoming@gitlab.example.com`. Example for Omnibus installs: @@ -218,7 +218,7 @@ incoming_email: #### Gmail -Example configuration for Gmail/G Suite. Assumes mailbox gitlab-incoming@gmail.com. +Example configuration for Gmail/G Suite. Assumes mailbox `gitlab-incoming@gmail.com`. Example for Omnibus installs: diff --git a/doc/administration/index.md b/doc/administration/index.md index 00c8863f200..650cb10a64a 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -2,7 +2,7 @@ description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- -# Administrator documentation **(CORE ONLY)** +# Administrator Docs **(CORE ONLY)** Learn how to administer your self-managed GitLab instance. @@ -64,6 +64,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. +- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification emails with S/MIME #### Customizing GitLab's appearance @@ -185,3 +186,13 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. +- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab + Support team. +- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch. +- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful + for troubleshooting Kubernetes-related issues. +- Useful links from the Support Team: + - [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html). + - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html). + - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html). + - [Strace zine](https://wizardzines.com/zines/strace/). diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index ab6a4646a71..825435deff9 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -14,4 +14,3 @@ Additionally you will be contacted by our team for further review which should h [6995]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6995 [ee]: https://about.gitlab.com/pricing/ - diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 8de7b0bc57e..16a193550a1 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -1,6 +1,7 @@ # PlantUML & GitLab -> [Introduced][ce-8537] in GitLab 8.16. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8537) in +> GitLab 8.16. When [PlantUML](http://plantuml.com) integration is enabled and configured in GitLab we are able to create simple diagrams in AsciiDoc and Markdown documents @@ -15,7 +16,9 @@ server that will generate the diagrams. With Docker, you can just run a container like this: -`docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat` +```sh +docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat +``` The **PlantUML URL** will be the hostname of the server running the container. @@ -26,7 +29,7 @@ own PlantUML server is easy in Debian/Ubuntu distributions using Tomcat. First you need to create a `plantuml.war` file from the source code: -``` +```sh sudo apt-get install graphviz openjdk-8-jdk git-core maven git clone https://github.com/plantuml/plantuml-server.git cd plantuml-server @@ -36,7 +39,7 @@ mvn package The above sequence of commands will generate a WAR file that can be deployed using Tomcat: -``` +```sh sudo apt-get install tomcat7 sudo cp target/plantuml.war /var/lib/tomcat7/webapps/plantuml.war sudo chown tomcat7:tomcat7 /var/lib/tomcat7/webapps/plantuml.war @@ -46,7 +49,7 @@ sudo service tomcat7 restart Once the Tomcat service restarts the PlantUML service will be ready and listening for requests on port 8080: -``` +```text http://localhost:8080/plantuml ``` @@ -57,9 +60,10 @@ you can change these defaults by editing the `/etc/tomcat7/server.xml` file. You need to enable PlantUML integration from Settings under Admin Area. To do that, login with an Admin account and do following: - - in GitLab go to **Admin Area**->**Settings**->**Integrations**->**PlantUML** - - check **Enable PlantUML** checkbox - - set the PlantUML instance as **PlantUML URL** +- In GitLab, go to **Admin Area > Settings > Integrations**. +- Expand the **PlantUML** section. +- Check **Enable PlantUML** checkbox. +- Set the PlantUML instance as **PlantUML URL**. ## Creating Diagrams @@ -68,33 +72,34 @@ our AsciiDoc snippets, wikis and repos using delimited blocks: - **Markdown** - <pre> - ```plantuml - Bob -> Alice : hello - Alice -> Bob : Go Away - ```</pre> + ~~~markdown + ```plantuml + Bob -> Alice : hello + Alice -> Bob : Go Away + ``` + ~~~ - **AsciiDoc** - ``` - [plantuml, format="png", id="myDiagram", width="200px"] - ---- - Bob->Alice : hello - Alice -> Bob : Go Away - ---- - ``` + ```text + [plantuml, format="png", id="myDiagram", width="200px"] + ---- + Bob->Alice : hello + Alice -> Bob : Go Away + ---- + ``` - **reStructuredText** - ``` - .. plantuml:: - :caption: Caption with **bold** and *italic* + ```text + .. plantuml:: + :caption: Caption with **bold** and *italic* - Bob -> Alice: hello - Alice -> Bob: Go Away - ``` + Bob -> Alice: hello + Alice -> Bob: Go Away + ``` - You can also use the `uml::` directive for compatibility with [sphinxcontrib-plantuml](https://pypi.org/project/sphinxcontrib-plantuml/), but please note that we currently only support the `caption` option. + You can also use the `uml::` directive for compatibility with [sphinxcontrib-plantuml](https://pypi.org/project/sphinxcontrib-plantuml/), but please note that we currently only support the `caption` option. The above blocks will be converted to an HTML img tag with source pointing to the PlantUML instance. If the PlantUML server is correctly configured, this should @@ -111,12 +116,10 @@ diagram delimiters `@startuml`/`@enduml` as these are replaced by the AsciiDoc ` Some parameters can be added to the AsciiDoc block definition: - - *format*: Can be either `png` or `svg`. Note that `svg` is not supported by - all browsers so use with care. The default is `png`. - - *id*: A CSS id added to the diagram HTML tag. - - *width*: Width attribute added to the img tag. - - *height*: Height attribute added to the img tag. +- *format*: Can be either `png` or `svg`. Note that `svg` is not supported by + all browsers so use with care. The default is `png`. +- *id*: A CSS id added to the diagram HTML tag. +- *width*: Width attribute added to the img tag. +- *height*: Height attribute added to the img tag. Markdown does not support any parameters and will always use PNG format. - -[ce-8537]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8537 diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index c34858cd0db..24c9cc0bea9 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -41,9 +41,9 @@ detail below. - Every session URL that is created has an authorization header that needs to be sent, to establish a `wss` connection. - The session URL is not exposed to the users in any way. GitLab holds all the state internally and proxies accordingly. -## Enabling and disabling terminal support +## Enabling and disabling terminal support -NOTE: **Note:** AWS Elastic Load Balancers (ELBs) do not support web sockets. +NOTE: **Note:** AWS Elastic Load Balancers (ELBs) do not support web sockets. AWS Application Load Balancers (ALBs) must be used if you want web terminals to work. See [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) for more information. diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index e1bbabb2878..f9be06c6daf 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -1,8 +1,8 @@ # Issue closing pattern **(CORE ONLY)** >**Note:** -This is the administration documentation. -There is a separate [user documentation] on issue closing pattern. +This is the administration documentation. There is a separate [user documentation](../user/project/issues/managing_issues.md#closing-issues-automatically) +on issue closing pattern. When a commit or merge request resolves one or more issues, it is possible to automatically have these issues closed when the commit or merge request lands @@ -13,8 +13,8 @@ in the project's default branch. In order to change the pattern you need to have access to the server that GitLab is installed on. -The default pattern can be located in [gitlab.yml.example] under the -"Automatic issue closing" section. +The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example) +under the "Automatic issue closing" section. > **Tip:** You are advised to use <http://rubular.com> to test the issue closing pattern. @@ -31,7 +31,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" ``` -1. [Reconfigure] GitLab for the changes to take effect. +1. [Reconfigure](restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. **For installations from source** @@ -42,9 +42,4 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" ``` -1. [Restart] GitLab for the changes to take effect. - -[gitlab.yml.example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example -[reconfigure]: restart_gitlab.md#omnibus-gitlab-reconfigure -[restart]: restart_gitlab.md#installations-from-source -[user documentation]: ../user/project/issues/managing_issues.md#closing-issues-automatically +1. [Restart](restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 9df7b2526e2..350cd5b7992 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -13,8 +13,6 @@ if you want to know how to disable it. To disable artifacts site-wide, follow the steps below. ---- - **In Omnibus installations:** 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -25,8 +23,6 @@ To disable artifacts site-wide, follow the steps below. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -49,8 +45,6 @@ this is done when the job succeeds, but can also be done on failure, or always, To change the location where the artifacts are stored locally, follow the steps below. ---- - **In Omnibus installations:** _The artifacts are stored by default in @@ -65,8 +59,6 @@ _The artifacts are stored by default in 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** _The artifacts are stored by default in @@ -102,7 +94,7 @@ If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need ### Object Storage Settings -For source installations the following settings are nested under `artifacts:` and then `object_store:`. On omnibus installs they are prefixed by `artifacts_object_store_`. +For source installations the following settings are nested under `artifacts:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `artifacts_object_store_`. | Setting | Description | Default | |---------|-------------|---------| @@ -123,6 +115,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -167,8 +160,6 @@ _The artifacts are stored by default in gitlab-rake gitlab:artifacts:migrate ``` ---- - **In installations from source:** _The artifacts are stored by default in @@ -207,8 +198,6 @@ right after that date passes. Artifacts are cleaned up by the To change the default schedule on which the artifacts are expired, follow the steps below. ---- - **In Omnibus installations:** 1. Edit `/etc/gitlab/gitlab.rb` and comment out or add the following line @@ -219,8 +208,6 @@ steps below. 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following @@ -240,8 +227,6 @@ steps below. To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-will-fail), you can flip the feature flag from a Rails console. ---- - **In Omnibus installations:** 1. Enter the Rails console: @@ -256,8 +241,6 @@ you can flip the feature flag from a Rails console. Feature.enable('ci_disable_validates_dependencies') ``` ---- - **In installations from source:** 1. Enter the Rails console: diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index 957916893d7..6a06eb240de 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -95,9 +95,9 @@ If job traces have already been archived into local storage, and you want to mig 1. Ensure [Object storage integration for Job Artifacts](job_artifacts.md#object-storage-settings) is enabled 1. Execute the following command - ```bash - gitlab-rake gitlab:traces:migrate - ``` + ```bash + gitlab-rake gitlab:traces:migrate + ``` ## How to remove job traces diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 5a2f389d298..9030c941cad 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -88,7 +88,7 @@ Introduced in GitLab 10.0, this file lives in It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} ``` This entry above shows an access to an internal endpoint to check whether an @@ -151,22 +151,29 @@ etc. For example: {"severity":"ERROR","time":"2018-11-23T15:42:11.647Z","exception":"Kubeclient::HttpError","error_code":null,"service":"Clusters::Applications::InstallService","app_id":2,"project_ids":[19],"group_ids":[],"message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"} ``` -## `githost.log` +## `git_json.log` -This file lives in `/var/log/gitlab/gitlab-rails/githost.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/githost.log` for +This file lives in `/var/log/gitlab/gitlab-rails/git_json.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/git_json.log` for installations from source. +NOTE: **Note:** +After 12.2, this file was renamed from `githost.log` to +`git_json.log` and stored in JSON format. + GitLab has to interact with Git repositories but in some rare cases something can go wrong and in this case you will know what exactly happened. This log file contains all failed requests from GitLab to Git repositories. In the majority of cases this file will be useful for developers only. For example: -``` -December 03, 2014 13:20 -> ERROR -> Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict - -error: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git' +```json +{ + "severity":"ERROR", + "time":"2019-07-19T22:16:12.528Z", + "correlation_id":"FeGxww5Hj64", + "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'" +} ``` ## `audit_json.log` @@ -236,7 +243,7 @@ I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory User clone/fetch activity using ssh transport appears in this log as `executing git command <gitaly-upload-pack...`. -## `unicorn\_stderr.log` +## `unicorn_stderr.log` This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for @@ -277,16 +284,16 @@ Introduced in GitLab 11.3. This file lives in `/var/log/gitlab/gitlab-rails/impo Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for installations from source. -Currently it logs the progress of project imports from the Bitbucket Server -importer. Future importers may use this file. - -## `auth.log` +## `auth.log` Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for installations from source. -It logs information whenever [Rack Attack] registers an abusive request. +This log records: + +- Information whenever [Rack Attack] registers an abusive request. +- Requests over the [Rate Limit] on raw endpoints. NOTE: **Note:** From [%12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/62756), user id and username are available on this log. @@ -305,6 +312,12 @@ GraphQL queries are recorded in that file. For example: {"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7} ``` +## `migrations.log` + +Introduced in GitLab 12.3. This file lives in `/var/log/gitlab/gitlab-rails/migrations.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/migrations.log` for +installations from source. + ## Reconfigure Logs Reconfigure log files live in `/var/log/gitlab/reconfigure` for Omnibus GitLab @@ -324,3 +337,4 @@ installations from source. [repocheck]: repository_checks.md [Rack Attack]: ../security/rack_attack.md +[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 99cd9051778..d52d865cec5 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -90,6 +90,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md new file mode 100644 index 00000000000..d445b68721d --- /dev/null +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -0,0 +1,36 @@ +# GitLab instance administration project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/56883) in GitLab 12.2. + +GitLab has been adding the ability for administrators to see insights into the health of +their GitLab instance. In order to surface this experience in a native way, similar to how +you would interact with an application deployed via GitLab, a base project called +"GitLab Instance Administration" with +[internal visibility](../../../public_access/public_access.md#internal-projects) will be +added under a group called "GitLab Instance Administrators" specifically created for +visualizing and configuring the monitoring of your GitLab instance. + +All administrators at the time of creation of the project and group will be added +as maintainers of the group and project, and as an admin, you'll be able to add new +members to the group in order to give them maintainer access to the project. + +This project will be used for self-monitoring your GitLab instance. + +## Connection to Prometheus + +The project will be automatically configured to connect to the +[internal Prometheus](../prometheus/index.md) instance if the Prometheus +instance is present (should be the case if GitLab was installed via Omnibus +and you haven't disabled it). + +If that's not the case or if you have an external Prometheus instance or an HA setup, +you should +[configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). + +## Taking action on Prometheus alerts **(ULTIMATE)** + +You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) +to the Prometheus config in order for GitLab to receive notifications of any alerts. + +Once the webhook is setup, you can +[take action on incoming alerts](../../../user/project/integrations/prometheus.md#taking-action-on-incidents-ultimate). diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index fa0459b24ff..2b3daec42bd 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -2,6 +2,9 @@ Explore our features to monitor your GitLab instance: +- [GitLab self-monitoring](gitlab_instance_administration_project/index.md): The + GitLab instance administration project helps to monitor the GitLab instance and + take action on alerts. - [Performance monitoring](performance/index.md): GitLab Performance Monitoring makes it possible to measure a wide variety of statistics of your instance. - [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. - [GitHub imports](github_imports.md): Monitor the health and progress of GitLab's GitHub importer with various Prometheus metrics. diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index ad2773de132..6bb2fe81b2c 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -12,9 +12,9 @@ hosts or use IP ranges: 1. Open `/etc/gitlab/gitlab.rb` and add or uncomment the following: - ```ruby - gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] - ``` + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` 1. Save the file and [reconfigure] GitLab for the changes to take effect. @@ -24,13 +24,13 @@ hosts or use IP ranges: 1. Edit `config/gitlab.yml`: - ```yaml - monitoring: - # by default only local IPs are allowed to access monitoring resources - ip_whitelist: - - 127.0.0.0/8 - - 192.168.0.1 - ``` + ```yaml + monitoring: + # by default only local IPs are allowed to access monitoring resources + ip_whitelist: + - 127.0.0.0/8 + - 192.168.0.1 + ``` 1. Save the file and [restart] GitLab for the changes to take effect. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 771584268d9..9a25cc04ee7 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -8,12 +8,8 @@ The minimum required settings you need to set are the InfluxDB host and port. Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the changes. ---- -  ---- - Finally, a restart of all GitLab processes is required for the changes to take effect: @@ -30,8 +26,6 @@ sudo service gitlab restart When any migrations are pending, the metrics are disabled until the migrations have been performed. ---- - Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 4dd0bbbe937..95be0d5fd88 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ # Grafana Configuration -[Grafana](https://grafana.org/) is a tool that allows you to visualize time +[Grafana](https://grafana.com/) is a tool that allows you to visualize time series metrics through graphs and dashboards. It supports several backend data stores, including InfluxDB. GitLab writes performance data to InfluxDB and Grafana will allow you to query to display useful graphs. @@ -118,6 +118,36 @@ If you have set up Grafana, you can enable a link to access it easily from the s 1. Click **Save changes**. 1. The new link will be available in the admin area under **Monitoring > Metrics Dashboard**. +## Security Update + +Users running GitLab version 12.0 or later should immediately upgrade to one of the following security releases due to a known vulnerability with the embedded Grafana dashboard: + +- 12.0.6 +- 12.1.6 + +After upgrading, the Grafana dashboard will be disabled and the location of your existing Grafana data will be changed from `/var/opt/gitlab/grafana/data/` to `/var/opt/gitlab/grafana/data.bak.#{Date.today}/`. + +To prevent the data from being relocated, you can run the following command prior to upgrading: + +```sh +echo "0" > /var/opt/gitlab/grafana/CVE_reset_status +``` + +To reinstate your old data, move it back into its original location: + +``` +sudo mv /var/opt/gitlab/grafana/data.bak.xxxx/ /var/opt/gitlab/grafana/data/ +``` + +However, you should **not** reinstate your old data _except_ under one of the following conditions: + +1. If you are certain that you changed your default admin password when you enabled Grafana +1. If you run GitLab in a private network, accessed only by trusted users, and your Grafana login page has not been exposed to the internet + +If you require access to your old Grafana data but do not meet one of these criteria, you may consider reinstating it temporarily, [exporting the dashboards](https://grafana.com/docs/reference/export_import/#exporting-a-dashboard) you need, then refreshing the data and [re-importing your dashboards](https://grafana.com/docs/reference/export_import/#importing-a-dashboard). Note that this poses a temporary vulnerability while your old Grafana data is in use, and the decision to do so should be weighed carefully with your need to access existing data and dashboards. + +For more information and further mitigation details, please refer to our [blog post on the security release](https://about.gitlab.com/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). + --- Read more on: diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex 2bf686f9017..d1187fd879a 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png Binary files differindex 7af6d401d1d..2c43201cbd0 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png Binary files differdeleted file mode 100644 index a55ce753101..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png Binary files differnew file mode 100644 index 00000000000..ecb2dffbf8d --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_redis_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png Binary files differnew file mode 100644 index 00000000000..210f80a713d --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_rugged_calls.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png Binary files differindex b3219b4fa94..6b571f4e85c 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png +++ b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png diff --git a/doc/administration/monitoring/performance/img/request_profile_result.png b/doc/administration/monitoring/performance/img/request_profile_result.png Binary files differindex 1b06e240fa0..9176a0b49fd 100644 --- a/doc/administration/monitoring/performance/img/request_profile_result.png +++ b/doc/administration/monitoring/performance/img/request_profile_result.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 95f497a1470..02f4b78bd60 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -8,16 +8,14 @@ activated, it looks as follows: It allows you to see (from left to right): - the current host serving the page -- the timing of the page (backend, frontend) - time taken and number of DB queries, click through for details of these queries  - time taken and number of [Gitaly] calls, click through for details of these calls  -- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). -  -- time taken and number of calls to Redis -- time taken and number of background jobs created by Sidekiq -- time taken and number of Ruby GC calls +- time taken and number of [Rugged] calls, click through for details of these calls +  +- time taken and number of Redis calls, click through for details of these calls +  On the far right is a request selector that allows you to view the same metrics (excluding the page timing and line profiler) for any requests made while the @@ -40,10 +38,7 @@ display it. You can toggle the Bar using the same shortcut. ---- -  ---- - [Gitaly]: ../../gitaly/index.md +[Rugged]: ../../high_availability/nfs.md#improving-nfs-performance-with-gitlab diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 726882fbb87..c32edb60f9d 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -5,9 +5,9 @@ 1. Grab the profiling token from **Monitoring > Requests Profiles** admin page (highlighted in a blue in the image below).  -1. Pass the header `X-Profile-Token: <token>` to the request you want to profile. You can use: +1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where `<mode>` can be `execution` or `memory`) to the request you want to profile. You can use: - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. - - `curl`. For example, `curl --header 'X-Profile-Token: <token>' https://gitlab.example.com/group/project`. + - `curl`. For example, `curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' https://gitlab.example.com/group/project`. 1. Once request is finished (which will take a little longer than usual), you can view the profiling output from **Monitoring > Requests Profiles** admin page.  diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 89501f20d99..4809bee0df6 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,7 +1,7 @@ # GitLab Prometheus metrics >**Note:** -Available since [Omnibus GitLab 9.3][29118]. For +Available since [Omnibus GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-ce/issues/29118). For installations from source you'll have to configure it yourself. To enable the GitLab Prometheus metrics: @@ -9,123 +9,202 @@ To enable the GitLab Prometheus metrics: 1. Log into GitLab as an administrator, and go to the Admin area. 1. Click on the gear, then click on Settings. 1. Find the `Metrics - Prometheus` section, and click `Enable Prometheus Metrics` -1. [Restart GitLab][restart] for the changes to take effect +1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect ## Collecting the metrics GitLab monitors its own internal service metrics, and makes them available at the -`/-/metrics` endpoint. Unlike other [Prometheus] exporters, in order to access -it, the client IP needs to be [included in a whitelist][whitelist]. +`/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, in order to access +it, the client IP needs to be [included in a whitelist](../ip_whitelist.md). For Omnibus and Chart installations, these metrics are automatically enabled and collected as of [GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1702). For source installations or earlier versions, these metrics will need to be enabled manually and collected by a Prometheus server. -## Unicorn Metrics available +## Metrics available The following metrics are available: -| Metric | Type | Since | Description | -|:--------------------------------- |:--------- |:----- |:----------- | -| db_ping_timeout | Gauge | 9.4 | Whether or not the last database ping timed out | -| db_ping_success | Gauge | 9.4 | Whether or not the last database ping succeeded | -| db_ping_latency_seconds | Gauge | 9.4 | Round trip time of the database ping | -| filesystem_access_latency_seconds | Gauge | 9.4 | Latency in accessing a specific filesystem | -| filesystem_accessible | Gauge | 9.4 | Whether or not a specific filesystem is accessible | -| filesystem_write_latency_seconds | Gauge | 9.4 | Write latency of a specific filesystem | -| filesystem_writable | Gauge | 9.4 | Whether or not the filesystem is writable | -| filesystem_read_latency_seconds | Gauge | 9.4 | Read latency of a specific filesystem | -| filesystem_readable | Gauge | 9.4 | Whether or not the filesystem is readable | -| http_requests_total | Counter | 9.4 | Rack request count | -| http_request_duration_seconds | Histogram | 9.4 | HTTP response time from rack middleware | -| pipelines_created_total | Counter | 9.4 | Counter of pipelines created | -| rack_uncaught_errors_total | Counter | 9.4 | Rack connections handling uncaught errors count | -| redis_ping_timeout | Gauge | 9.4 | Whether or not the last redis ping timed out | -| redis_ping_success | Gauge | 9.4 | Whether or not the last redis ping succeeded | -| redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | -| user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | -| upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | -| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | -| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | -| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | -| unicorn_workers | Gauge | 12.0 | The number of Unicorn workers | +| Metric | Type | Since | Description | Labels | +|:---------------------------------------------------------------|:----------|-----------------------:|:----------------------------------------------------------------------------------------------------|:----------------------------------------------------| +| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering markdown into HTML when cached output exists | controller, action | +| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering markdown into HTML when cached outupt does not exist | controller, action | +| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | controller, action | +| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | controller, action, operation | +| `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | +| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | controller, action, module, method | +| `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | +| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT | | +| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (gitlab_transaction_* metrics) | | +| `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | +| `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | +| `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | controller, action | +| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | controller, action | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | controller, action | +| `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for api /jobs/request | | +| `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for api /jobs/request | | +| `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for api /jobs/request | | +| `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for api /jobs/request | | +| `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | +| `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for etag cache hit. | endpoint | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for etag cache miss - header missing | endpoint | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for etag cache miss - key not found | endpoint | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for etag middleware accessed | endpoint | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for etag cache miss - resource changed | endpoint | +| `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | +| `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | +| `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | +| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | branch | +| `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | +| `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | +| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for recieved emails | handler | +| `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | +| `gitlab_transaction_event_remove_branch_total` | Counter | 9.4 | Counter when a branch is removed for any repository | | +| `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | +| `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | +| `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of sidekiq exceptions | | +| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | projects_without_jid_count, projects_with_jid_count | +| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for api /jobs/request/:id | | +| `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new redis connections | | +| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | +| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | controller, action | +| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | controller, action, view | +| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | controller, action, view | +| `http_requests_total` | Counter | 9.4 | Rack request count | method | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | method, status | +| `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | +| `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | +| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in | | +| `upload_file_does_not_exist` | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | | +| `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | +| `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | + +## Metrics controlled by a feature flag + +The following metrics can be controlled by feature flags: + +| Metric | Feature Flag | +|:---------------------------------------------------------------|:-------------------------------------------------------------------| +| `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | +| `gitlab_transaction_allocated_memory_bytes` | `prometheus_metrics_transaction_allocated_memory` | +| `gitlab_transaction_event_build_found_total` | `prometheus_transaction_event_build_found_total` | +| `gitlab_transaction_event_build_invalid_total` | `prometheus_transaction_event_build_invalid_total` | +| `gitlab_transaction_event_build_not_found_cached_total` | `prometheus_transaction_event_build_not_found_cached_total` | +| `gitlab_transaction_event_build_not_found_total` | `prometheus_transaction_event_build_not_found_total` | +| `gitlab_transaction_event_change_default_branch_total` | `prometheus_transaction_event_change_default_branch_total` | +| `gitlab_transaction_event_create_repository_total` | `prometheus_transaction_event_create_repository_total` | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | `prometheus_transaction_event_etag_caching_cache_hit_total` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | `prometheus_transaction_event_etag_caching_header_missing_total` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | `prometheus_transaction_event_etag_caching_key_not_found_total` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | `prometheus_transaction_event_etag_caching_middleware_used_total` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | `prometheus_transaction_event_etag_caching_resource_changed_total` | +| `gitlab_transaction_event_fork_repository_total` | `prometheus_transaction_event_fork_repository_total` | +| `gitlab_transaction_event_import_repository_total` | `prometheus_transaction_event_import_repository_total` | +| `gitlab_transaction_event_push_branch_total` | `prometheus_transaction_event_push_branch_total` | +| `gitlab_transaction_event_push_commit_total` | `prometheus_transaction_event_push_commit_total` | +| `gitlab_transaction_event_push_tag_total` | `prometheus_transaction_event_push_tag_total` | +| `gitlab_transaction_event_rails_exception_total` | `prometheus_transaction_event_rails_exception_total` | +| `gitlab_transaction_event_receive_email_total` | `prometheus_transaction_event_receive_email_total` | +| `gitlab_transaction_event_remote_mirrors_failed_total` | `prometheus_transaction_event_remote_mirrors_failed_total` | +| `gitlab_transaction_event_remote_mirrors_finished_total` | `prometheus_transaction_event_remote_mirrors_finished_total` | +| `gitlab_transaction_event_remote_mirrors_running_total` | `prometheus_transaction_event_remote_mirrors_running_total` | +| `gitlab_transaction_event_remove_branch_total` | `prometheus_transaction_event_remove_branch_total` | +| `gitlab_transaction_event_remove_repository_total` | `prometheus_transaction_event_remove_repository_total` | +| `gitlab_transaction_event_remove_tag_total` | `prometheus_transaction_event_remove_tag_total` | +| `gitlab_transaction_event_sidekiq_exception_total` | `prometheus_transaction_event_sidekiq_exception_total` | +| `gitlab_transaction_event_stuck_import_jobs_total` | `prometheus_transaction_event_stuck_import_jobs_total` | +| `gitlab_transaction_event_update_build_total` | `prometheus_transaction_event_update_build_total` | +| `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | ## Sidekiq Metrics available for Geo **(PREMIUM)** Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. -| Metric | Type | Since | Description | Labels | -|:-------------------------------------------- |:------- |:----- |:----------- |:------ | -| geo_db_replication_lag_seconds | Gauge | 10.2 | Database replication lag (seconds) | url -| geo_repositories | Gauge | 10.2 | Total number of repositories available on primary | url -| geo_repositories_synced | Gauge | 10.2 | Number of repositories synced on secondary | url -| geo_repositories_failed | Gauge | 10.2 | Number of repositories failed to sync on secondary | url -| geo_lfs_objects | Gauge | 10.2 | Total number of LFS objects available on primary | url -| geo_lfs_objects_synced | Gauge | 10.2 | Number of LFS objects synced on secondary | url -| geo_lfs_objects_failed | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url -| geo_attachments | Gauge | 10.2 | Total number of file attachments available on primary | url -| geo_attachments_synced | Gauge | 10.2 | Number of attachments synced on secondary | url -| geo_attachments_failed | Gauge | 10.2 | Number of attachments failed to sync on secondary | url -| geo_last_event_id | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url -| geo_last_event_timestamp | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url -| geo_cursor_last_event_id | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url -| geo_cursor_last_event_timestamp | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url -| geo_status_failed_total | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url -| geo_last_successful_status_check_timestamp | Gauge | 10.2 | Last timestamp when the status was successfully updated | url -| geo_lfs_objects_synced_missing_on_primary | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url -| geo_job_artifacts_synced_missing_on_primary | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url -| geo_attachments_synced_missing_on_primary | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url -| geo_repositories_checksummed_count | Gauge | 10.7 | Number of repositories checksummed on primary | url -| geo_repositories_checksum_failed_count | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url -| geo_wikis_checksummed_count | Gauge | 10.7 | Number of wikis checksummed on primary | url -| geo_wikis_checksum_failed_count | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url -| geo_repositories_verified_count | Gauge | 10.7 | Number of repositories verified on secondary | url -| geo_repositories_verification_failed_count | Gauge | 10.7 | Number of repositories failed to verify on secondary | url -| geo_repositories_checksum_mismatch_count | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url -| geo_wikis_verified_count | Gauge | 10.7 | Number of wikis verified on secondary | url -| geo_wikis_verification_failed_count | Gauge | 10.7 | Number of wikis failed to verify on secondary | url -| geo_wikis_checksum_mismatch_count | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url -| geo_repositories_checked_count | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url -| geo_repositories_checked_failed_count | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url -| geo_repositories_retrying_verification_count | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url -| geo_wikis_retrying_verification_count | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url +| Metric | Type | Since | Description | Labels | +|:---------------------------------------------- |:------- |:----- |:----------- |:------ | +| `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | url | +| `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | url | +| `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | url | +| `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | url | +| `geo_lfs_objects` | Gauge | 10.2 | Total number of LFS objects available on primary | url | +| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of LFS objects synced on secondary | url | +| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url | +| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | url | +| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | url | +| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | url | +| `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url | +| `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url | +| `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url | +| `geo_cursor_last_event_timestamp` | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url | +| `geo_status_failed_total` | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url | +| `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | url | +| `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url | +| `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url | +| `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url | +| `geo_repositories_checksummed_count` | Gauge | 10.7 | Number of repositories checksummed on primary | url | +| `geo_repositories_checksum_failed_count` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url | +| `geo_wikis_checksummed_count` | Gauge | 10.7 | Number of wikis checksummed on primary | url | +| `geo_wikis_checksum_failed_count` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url | +| `geo_repositories_verified_count` | Gauge | 10.7 | Number of repositories verified on secondary | url | +| `geo_repositories_verification_failed_count` | Gauge | 10.7 | Number of repositories failed to verify on secondary | url | +| `geo_repositories_checksum_mismatch_count` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url | +| `geo_wikis_verified_count` | Gauge | 10.7 | Number of wikis verified on secondary | url | +| `geo_wikis_verification_failed_count` | Gauge | 10.7 | Number of wikis failed to verify on secondary | url | +| `geo_wikis_checksum_mismatch_count` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url | +| `geo_repositories_checked_count` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url | +| `geo_repositories_checked_failed_count` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url | +| `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url | +| `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url | ### Ruby metrics Some basic Ruby runtime metrics are available: -| Metric | Type | Since | Description | -|:-------------------------------------- |:--------- |:----- |:----------- | -| ruby_gc_duration_seconds_total | Counter | 11.1 | Time spent by Ruby in GC | -| ruby_gc_stat_... | Gauge | 11.1 | Various metrics from [GC.stat] | -| ruby_file_descriptors | Gauge | 11.1 | File descriptors per process | -| ruby_memory_bytes | Gauge | 11.1 | Memory usage by process | -| ruby_sampler_duration_seconds_total | Counter | 11.1 | Time spent collecting stats | -| ruby_process_cpu_seconds_total | Gauge | 12.0 | Total amount of CPU time per process | -| ruby_process_max_fds | Gauge | 12.0 | Maximum number of open file descriptors per process | -| ruby_process_resident_memory_bytes | Gauge | 12.0 | Memory usage by process, measured in bytes | -| ruby_process_start_time_seconds | Gauge | 12.0 | UNIX timestamp of process start time | +| Metric | Type | Since | Description | +|:------------------------------------ |:--------- |:----- |:----------- | +| `ruby_gc_duration_seconds` | Counter | 11.1 | Time spent by Ruby in GC | +| `ruby_gc_stat_...` | Gauge | 11.1 | Various metrics from [GC.stat] | +| `ruby_file_descriptors` | Gauge | 11.1 | File descriptors per process | +| `ruby_memory_bytes` | Gauge | 11.1 | Memory usage by process | +| `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | +| `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | +| `ruby_process_max_fds` | Gauge | 12.0 | Maximum number of open file descriptors per process | +| `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process, measured in bytes | +| `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | -[GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat +[GC.stat]: https://ruby-doc.org/core-2.6.3/GC.html#method-c-stat + +## Unicorn Metrics + +Unicorn specific metrics, when Unicorn is used. + +| Metric | Type | Since | Description | +|:-----------------------------|:------|:------|:---------------------------------------------------| +| `unicorn_active_connections` | Gauge | 11.0 | The number of active Unicorn connections (workers) | +| `unicorn_queued_connections` | Gauge | 11.0 | The number of queued Unicorn connections | +| `unicorn_workers` | Gauge | 12.0 | The number of Unicorn workers | ## Puma Metrics **(EXPERIMENTAL)** -When Puma is used instead of Unicorn, following metrics are available: - -| Metric | Type | Since | Description | -|:-------------------------------------------- |:------- |:----- |:----------- | -| puma_workers | Gauge | 12.0 | Total number of workers | -| puma_running_workers | Gauge | 12.0 | Number of booted workers | -| puma_stale_workers | Gauge | 12.0 | Number of old workers | -| puma_phase | Gauge | 12.0 | Phase number (increased during phased restarts) | -| puma_running | Gauge | 12.0 | Number of running threads | -| puma_queued_connections | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | -| puma_active_connections | Gauge | 12.0 | Number of threads processing a request | -| puma_pool_capacity | Gauge | 12.0 | Number of requests the worker is capable of taking right now | -| puma_max_threads | Gauge | 12.0 | Maximum number of worker threads | -| puma_idle_threads | Gauge | 12.0 | Number of spawned threads which are not processing a request | -| rack_state_total | Gauge | 12.0 | Number of requests in a given rack state | -| puma_killer_terminations_total | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | +When Puma is used instead of Unicorn, the following metrics are available: + +| Metric | Type | Since | Description | +|:---------------------------------------------- |:------- |:----- |:----------- | +| `puma_workers` | Gauge | 12.0 | Total number of workers | +| `puma_running_workers` | Gauge | 12.0 | Number of booted workers | +| `puma_stale_workers` | Gauge | 12.0 | Number of old workers | +| `puma_running` | Gauge | 12.0 | Number of running threads | +| `puma_queued_connections` | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | +| `puma_active_connections` | Gauge | 12.0 | Number of threads processing a request | +| `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | +| `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | +| `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | +| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | ## Metrics shared directory @@ -142,9 +221,3 @@ If GitLab is installed using Omnibus and `tmpfs` is available then metrics directory will be automatically configured. [← Back to the main Prometheus page](index.md) - -[29118]: https://gitlab.com/gitlab-org/gitlab-ce/issues/29118 -[Prometheus]: https://prometheus.io -[restart]: ../../restart_gitlab.md#omnibus-gitlab-restart -[whitelist]: ../ip_whitelist.md -[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md index f68b03d1ade..9aa4dfa5ab7 100644 --- a/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_monitor_exporter.md @@ -12,9 +12,9 @@ To enable the GitLab monitor exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - gitlab_monitor['enable'] = true - ``` + ```ruby + gitlab_monitor['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index ce65d77274b..c8968c51393 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -39,9 +39,9 @@ To disable Prometheus and all of its exporters, as well as any added in the futu 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `false`: - ```ruby - prometheus_monitoring['enable'] = false - ``` + ```ruby + prometheus_monitoring['enable'] = false + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect. @@ -61,19 +61,19 @@ To change the address/port that Prometheus listens on: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line: - ```ruby - prometheus['listen_address'] = 'localhost:9090' - ``` + ```ruby + prometheus['listen_address'] = 'localhost:9090' + ``` - Replace `localhost:9090` with the address/port you want Prometheus to - listen on. If you would like to allow access to Prometheus to hosts other - than `localhost`, leave out the host, or use `0.0.0.0` to allow public access: + Replace `localhost:9090` with the address/port you want Prometheus to + listen on. If you would like to allow access to Prometheus to hosts other + than `localhost`, leave out the host, or use `0.0.0.0` to allow public access: - ```ruby - prometheus['listen_address'] = ':9090' - # or - prometheus['listen_address'] = '0.0.0.0:9090' - ``` + ```ruby + prometheus['listen_address'] = ':9090' + # or + prometheus['listen_address'] = '0.0.0.0:9090' + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect @@ -90,22 +90,22 @@ To use an external Prometheus server: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Disable the bundled Prometheus: - ```ruby - prometheus['enable'] = false - ``` + ```ruby + prometheus['enable'] = false + ``` 1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example: - ```ruby - gitlab_monitor['listen_address'] = '0.0.0.0' - sidekiq['listen_address'] = '0.0.0.0' - gitlab_monitor['listen_port'] = '9168' - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" - ``` + ```ruby + gitlab_monitor['listen_address'] = '0.0.0.0' + sidekiq['listen_address'] = '0.0.0.0' + gitlab_monitor['listen_port'] = '9168' + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" + ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). 1. Add the Prometheus server IP address to the [monitoring IP whitelist](../ip_whitelist.html). For example: @@ -117,14 +117,14 @@ To use an external Prometheus server: 1. To scrape nginx metrics, you'll also need to configure nginx to allow the Prometheus server IP. For example: - ```ruby - nginx['status']['options'] = { - "server_tokens" => "off", - "access_log" => "off", - "allow" => "192.168.0.1", - "deny" => "all", - } - ``` + ```ruby + nginx['status']['options'] = { + "server_tokens" => "off", + "access_log" => "off", + "allow" => "192.168.0.1", + "deny" => "all", + } + ``` 1. [Reconfigure GitLab][reconfigure] to apply the changes 1. Edit the Prometheus server's configuration file. @@ -132,19 +132,59 @@ To use an external Prometheus server: [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E). For example, a sample snippet using `static_configs`: - ```yaml - scrape_configs: - - job_name: 'gitlab_exporters' - static_configs: - - targets: ['1.1.1.1:9168', '1.1.1.1:9236', '1.1.1.1:9236', '1.1.1.1:9100', '1.1.1.1:9121', '1.1.1.1:9187'] - - - job_name: 'gitlab_metrics' - metrics_path: /-/metrics - static_configs: - - targets: ['1.1.1.1:443'] - ``` - -1. Restart the Prometheus server. + ```yaml + scrape_configs: + - job_name: nginx + static_configs: + - targets: + - 1.1.1.1:8060 + - job_name: redis + static_configs: + - targets: + - 1.1.1.1:9121 + - job_name: postgres + static_configs: + - targets: + - 1.1.1.1:9187 + - job_name: node + static_configs: + - targets: + - 1.1.1.1:9100 + - job_name: gitlab-workhorse + static_configs: + - targets: + - 1.1.1.1:9229 + - job_name: gitlab-rails + metrics_path: "/-/metrics" + static_configs: + - targets: + - 1.1.1.1:8080 + - job_name: gitlab-sidekiq + static_configs: + - targets: + - 1.1.1.1:8082 + - job_name: gitlab_monitor_database + metrics_path: "/database" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitlab_monitor_sidekiq + metrics_path: "/sidekiq" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitlab_monitor_process + metrics_path: "/process" + static_configs: + - targets: + - 1.1.1.1:9168 + - job_name: gitaly + static_configs: + - targets: + - 1.1.1.1:9236 + ``` + +1. Reload the Prometheus server. ## Viewing performance metrics @@ -241,9 +281,9 @@ To disable the monitoring of Kubernetes: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Add or find and uncomment the following line and set it to `false`: - ```ruby - prometheus['monitor_kubernetes'] = false - ``` + ```ruby + prometheus['monitor_kubernetes'] = false + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index aef7758a88f..bcacfaa3be5 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -13,9 +13,9 @@ To enable the node exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - node_exporter['enable'] = true - ``` + ```ruby + node_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index d76834fdbea..6183594c69c 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -12,9 +12,9 @@ To enable the PgBouncer exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - pgbouncer_exporter['enable'] = true - ``` + ```ruby + pgbouncer_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 8e2d3162f88..3ad15b65497 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -12,9 +12,9 @@ To enable the postgres exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - postgres_exporter['enable'] = true - ``` + ```ruby + postgres_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index d54d409dbb6..1520115a38d 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -13,9 +13,9 @@ To enable the Redis exporter: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line, making sure it's set to `true`: - ```ruby - redis_exporter['enable'] = true - ``` + ```ruby + redis_exporter['enable'] = true + ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index b329abdca08..e787af798bc 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -71,10 +71,10 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. -> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. -> **Warning:** Do not disable writes until SSH is confirmed to be working +CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. In the case of lookup failures (which are common), the `authorized_keys` @@ -117,81 +117,81 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. First, download the package and install the required packages: - ``` - sudo su - - cd /tmp - curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz - tar xzvf openssh-7.5p1.tar.gz - yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel - ``` + ``` + sudo su - + cd /tmp + curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz + tar xzvf openssh-7.5p1.tar.gz + yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel + ``` 1. Prepare the build by copying files to the right place: - ``` - mkdir -p /root/rpmbuild/{SOURCES,SPECS} - cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ - cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ - cd /root/rpmbuild/SPECS - ``` + ``` + mkdir -p /root/rpmbuild/{SOURCES,SPECS} + cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ + cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ + cd /root/rpmbuild/SPECS + ``` 1. Next, set the spec settings properly: - ``` - sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec - sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec - sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec - ``` + ``` + sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec + sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec + sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec + ``` 1. Build the RPMs: - ``` - rpmbuild -bb openssh.spec - ``` + ``` + rpmbuild -bb openssh.spec + ``` 1. Ensure the RPMs were built: - ``` - ls -al /root/rpmbuild/RPMS/x86_64/ - ``` + ``` + ls -al /root/rpmbuild/RPMS/x86_64/ + ``` - You should see something as the following: + You should see something as the following: - ``` - total 1324 - drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . - drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. - -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm - ``` + ``` + total 1324 + drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . + drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. + -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm + -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm + -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm + -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm + ``` 1. Install the packages. OpenSSH packages will replace `/etc/pam.d/sshd` with its own version, which may prevent users from logging in, so be sure that the file is backed up and restored after installation: - ``` - timestamp=$(date +%s) - cp /etc/pam.d/sshd pam-ssh-conf-$timestamp - rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm - yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd - ``` + ``` + timestamp=$(date +%s) + cp /etc/pam.d/sshd pam-ssh-conf-$timestamp + rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm + yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd + ``` 1. Verify the installed version. In another window, attempt to login to the server: - ``` - ssh -v <your-centos-machine> - ``` + ``` + ssh -v <your-centos-machine> + ``` - You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" + You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" - If not, you may need to restart sshd (e.g. `systemctl restart sshd.service`). + If not, you may need to restart sshd (e.g. `systemctl restart sshd.service`). -1. *IMPORTANT!* Open a new SSH session to your server before exiting to make - sure everything is working! If you need to downgrade, simple install the - older package: +1. *IMPORTANT!* Open a new SSH session to your server before exiting to make + sure everything is working! If you need to downgrade, simple install the + older package: - ``` - # Only run this if you run into a problem logging in - yum downgrade openssh-server openssh openssh-clients - ``` + ``` + # Only run this if you run into a problem logging in + yum downgrade openssh-server openssh openssh-clients + ``` diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index c0c242733a2..b5922d9d99d 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -78,34 +78,37 @@ executed, and then read the same 1,000 files. [repository storage path](../repository_storage_paths.md). 1. Create a temporary directory for the test so it's easy to remove the files later: - ```sh - mkdir test; cd test - ``` + ```sh + mkdir test; cd test + ``` + 1. Run the command: - ```sh - time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done - ``` -1. To benchmark read performance, run the command: + ```sh + time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done + ``` - ```sh - time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done - ``` -1. Remove the test files: +1. To benchmark read performance, run the command: ```sh - cd ../; rm -rf test + time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done ``` +1. Remove the test files: + + ```sh + cd ../; rm -rf test + ``` + The output of the `time for ...` commands will look similar to the following. The important metric is the `real` time. ```sh $ time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done -real 0m0.116s -user 0m0.025s -sys 0m0.091s +real 0m0.116s +user 0m0.025s +sys 0m0.091s $ time for i in {0..1000}; do cat "test${i}.txt" > /dev/null; done diff --git a/doc/administration/operations/img/sidekiq-cluster.png b/doc/administration/operations/img/sidekiq-cluster.png Binary files differindex 4eb1849010e..3899385eb8f 100644 --- a/doc/administration/operations/img/sidekiq-cluster.png +++ b/doc/administration/operations/img/sidekiq-cluster.png diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index ae67d7f08d6..8178cb243f3 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -69,7 +69,7 @@ unicorn['worker_memory_limit_min'] = "400 * 1 << 20" unicorn['worker_memory_limit_max'] = "650 * 1 << 20" ``` -Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MIN` +Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MAX` [environment variables](../environment_variables.md). This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. diff --git a/doc/administration/packages.md b/doc/administration/packages.md index c0f8777a8c0..1628b6d6f91 100644 --- a/doc/administration/packages.md +++ b/doc/administration/packages.md @@ -55,7 +55,7 @@ local location or even use object storage. The packages for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/packages/` and for source -installations under `shared/packages/` (relative to the git homedir). +installations under `shared/packages/` (relative to the Git homedir). To change the local storage path: **Omnibus GitLab installations** diff --git a/doc/administration/pages/img/lets_encrypt_integration_v12_1.png b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png Binary files differnew file mode 100644 index 00000000000..0f3ca25ce55 --- /dev/null +++ b/doc/administration/pages/img/lets_encrypt_integration_v12_1.png diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 102656b01ec..774e7056845 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -124,9 +124,9 @@ The Pages daemon doesn't listen to the outside world. 1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`: - ```shell - pages_external_url 'http://example.io' - ``` + ```shell + pages_external_url 'http://example.io' + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -149,16 +149,16 @@ outside world. 1. Place the certificate and key inside `/etc/gitlab/ssl` 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: - ```shell - pages_external_url 'https://example.io' + ```shell + pages_external_url 'https://example.io' - pages_nginx['redirect_http_to_https'] = true - pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" - pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key" - ``` + pages_nginx['redirect_http_to_https'] = true + pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" + pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key" + ``` - where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, - respectively. + where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, + respectively. 1. [Reconfigure GitLab][reconfigure]. @@ -171,9 +171,9 @@ behavior: 1. Edit `/etc/gitlab/gitlab.rb`. 1. Set the `inplace_chroot` to `true` for GitLab Pages: - ```shell - gitlab_pages['inplace_chroot'] = true - ``` + ```shell + gitlab_pages['inplace_chroot'] = true + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -206,16 +206,16 @@ world. Custom domains are supported, but no TLS. 1. Edit `/etc/gitlab/gitlab.rb`: - ```shell - pages_external_url "http://example.io" - nginx['listen_addresses'] = ['192.0.2.1'] - pages_nginx['enable'] = false - gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] - ``` + ```shell + pages_external_url "http://example.io" + nginx['listen_addresses'] = ['192.0.2.1'] + pages_nginx['enable'] = false + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] + ``` - where `192.0.2.1` is the primary IP address that GitLab is listening to and - `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon - listens on. If you don't have IPv6, you can omit the IPv6 address. + where `192.0.2.1` is the primary IP address that GitLab is listening to and + `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon + listens on. If you don't have IPv6, you can omit the IPv6 address. 1. [Reconfigure GitLab][reconfigure]. @@ -237,26 +237,26 @@ world. Custom domains and TLS are supported. 1. Edit `/etc/gitlab/gitlab.rb`: - ```shell - pages_external_url "https://example.io" - nginx['listen_addresses'] = ['192.0.2.1'] - pages_nginx['enable'] = false - gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" - gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" - gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] - gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001::2]:443'] - ``` + ```shell + pages_external_url "https://example.io" + nginx['listen_addresses'] = ['192.0.2.1'] + pages_nginx['enable'] = false + gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" + gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] + gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001::2]:443'] + ``` - where `192.0.2.1` is the primary IP address that GitLab is listening to and - `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon - listens on. If you don't have IPv6, you can omit the IPv6 address. + where `192.0.2.1` is the primary IP address that GitLab is listening to and + `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon + listens on. If you don't have IPv6, you can omit the IPv6 address. 1. [Reconfigure GitLab][reconfigure]. ### Custom domain verification To prevent malicious users from hijacking domains that don't belong to them, -GitLab supports [custom domain verification](../../user/project/pages/getting_started_part_three.md#dns-txt-record). +GitLab supports [custom domain verification](../../user/project/pages/custom_domains_ssl_tls_certification/index.md#steps). When adding a custom domain, users will be required to prove they own it by adding a GitLab-controlled verification code to the DNS records for that domain. @@ -265,6 +265,23 @@ verification requirement. Navigate to `Admin area ➔ Settings` and uncheck **Require users to prove ownership of custom domains** in the Pages section. This setting is enabled by default. +### Let's Encrypt integration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28996) in GitLab 12.1. + +[GitLab Pages' Let's Encrypt integration](../../user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md) +allows users to add Let's Encrypt SSL certificates for GitLab Pages +sites served under a custom domain. + +To enable it, you'll need to: + +1. Choose an email on which you will recieve notifications about expiring domains. +1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. +1. Enter the email for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. +1. Click **Save changes**. + + + ### Access control > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/33422) in GitLab 11.5. @@ -287,9 +304,9 @@ Pages access control is disabled by default. To enable it: 1. Enable it in `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_pages['access_control'] = true - ``` + ```ruby + gitlab_pages['access_control'] = true + ``` 1. [Reconfigure GitLab][reconfigure]. 1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control-core-only). @@ -302,9 +319,9 @@ pages: 1. Configure in `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_pages['http_proxy'] = 'http://example:8080' - ``` + ```ruby + gitlab_pages['http_proxy'] = 'http://example:8080' + ``` 1. [Reconfigure Gitlab][reconfigure] for the changes to take effect. @@ -319,9 +336,9 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. If you wish to make it log events with level `DEBUG` you must configure this in `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['log_verbose'] = true - ``` + ```shell + gitlab_pages['log_verbose'] = true + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -334,9 +351,9 @@ are stored. If you wish to store them in another location you must set it up in `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_rails['pages_path'] = "/mnt/storage/pages" - ``` + ```shell + gitlab_rails['pages_path'] = "/mnt/storage/pages" + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -347,19 +364,19 @@ Omnibus GitLab 11.1. 1. By default the listener is configured to listen for requests on `localhost:8090`. - If you wish to disable it you must configure this in - `/etc/gitlab/gitlab.rb`: + If you wish to disable it you must configure this in + `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['listen_proxy'] = nil - ``` + ```shell + gitlab_pages['listen_proxy'] = nil + ``` - If you wish to make it listen on a different port you must configure this also in - `/etc/gitlab/gitlab.rb`: + If you wish to make it listen on a different port you must configure this also in + `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['listen_proxy'] = "localhost:10080" - ``` + ```shell + gitlab_pages['listen_proxy'] = "localhost:10080" + ``` 1. [Reconfigure GitLab][reconfigure]. @@ -381,28 +398,29 @@ Follow the steps below to configure GitLab Pages in a separate server. 1. On `app2` install GitLab omnibus and modify `/etc/gitlab/gitlab.rb` this way: - ```shell - external_url 'http://<ip-address-of-the-server>' - pages_external_url "http://<your-pages-domain>" - postgresql['enable'] = false - redis['enable'] = false - prometheus['enable'] = false - unicorn['enable'] = false - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - gitaly['enable'] = false - alertmanager['enable'] = false - node_exporter['enable'] = false - gitlab_rails['auto_migrate'] = false - ``` + ```shell + external_url 'http://<ip-address-of-the-server>' + pages_external_url "http://<your-pages-domain>" + postgresql['enable'] = false + redis['enable'] = false + prometheus['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + gitaly['enable'] = false + alertmanager['enable'] = false + node_exporter['enable'] = false + gitlab_rails['auto_migrate'] = false + ``` + 1. Run `sudo gitlab-ctl reconfigure`. 1. On `app1` apply the following changes to `/etc/gitlab/gitlab.rb`: - ```shell - gitlab_pages['enable'] = false - pages_external_url "http://<your-pages-domain>" - gitlab_rails['pages_path'] = "/mnt/pages" - ``` + ```shell + gitlab_pages['enable'] = false + pages_external_url "http://<your-pages-domain>" + gitlab_rails['pages_path'] = "/mnt/pages" + ``` 1. Run `sudo gitlab-ctl reconfigure`. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 2100f7cd707..c77a1a9638f 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -24,8 +24,6 @@ SNI and exposes pages using HTTP2 by default. You are encouraged to read its [README][pages-readme] to fully understand how it works. ---- - In the case of [custom domains](#custom-domains) (but not [wildcard domains](#wildcard-domains)), the Pages daemon needs to listen on ports `80` and/or `443`. For that reason, there is some flexibility in the way @@ -92,8 +90,6 @@ since that is needed in all configurations. - [Wildcard DNS setup](#dns-configuration) ---- - URL scheme: `http://page.example.io` This is the minimum setup that you can use Pages with. It is the base for all @@ -102,90 +98,89 @@ The Pages daemon doesn't listen to the outside world. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. Go to the GitLab installation directory: - ```bash - cd /home/git/gitlab - ``` + ```bash + cd /home/git/gitlab + ``` 1. Edit `gitlab.yml` and under the `pages` setting, set `enabled` to `true` and the `host` to the FQDN under which GitLab Pages will be served: - ```yaml - ## GitLab Pages - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```yaml + ## GitLab Pages + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 80 - https: false - ``` + host: example.io + port: 80 + https: false + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the `-pages-domain` must match the `host` setting that you set above. - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090" - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090" + ``` 1. Copy the `gitlab-pages` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf + ``` 1. Restart NGINX 1. [Restart GitLab][restart] ### Wildcard domains with TLS support -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Wildcard TLS certificate -> -> --- -> -> URL scheme: `https://page.example.io` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Wildcard TLS certificate + +URL scheme: `https://page.example.io` Nginx will proxy all requests to the daemon. Pages daemon doesn't listen to the outside world. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. In `gitlab.yml`, set the port to `443` and https to `true`: - ```bash - ## GitLab Pages - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```bash + ## GitLab Pages + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 443 - https: true - ``` + host: example.io + port: 443 + https: true + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the @@ -193,17 +188,17 @@ outside world. The `-root-cert` and `-root-key` settings are the wildcard TLS certificates of the `example.io` domain: - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key + ``` 1. Copy the `gitlab-pages-ssl` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf + ``` 1. Restart NGINX 1. [Restart GitLab][restart] @@ -217,13 +212,12 @@ that without TLS certificates. ### Custom domains -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Secondary IP -> -> --- -> -> URL scheme: `http://page.example.io` and `http://domain.com` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Secondary IP + +URL scheme: `http://page.example.io` and `http://domain.com` In that case, the pages daemon is running, Nginx still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -231,48 +225,48 @@ world. Custom domains are supported, but no TLS. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. Edit `gitlab.yml` to look like the example below. You need to change the `host` to the FQDN under which GitLab Pages will be served. Set `external_http` to the secondary IP on which the pages daemon will listen for connections: - ```yaml - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```yaml + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 80 - https: false + host: example.io + port: 80 + https: false - external_http: 192.0.2.2:80 - ``` + external_http: 192.0.2.2:80 + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the `-pages-domain` and `-listen-http` must match the `host` and `external_http` settings that you set above respectively: - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80" - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80" + ``` 1. Copy the `gitlab-pages-ssl` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages /etc/nginx/sites-available/gitlab-pages.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf + ``` 1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace `0.0.0.0` with `192.0.2.1`, where `192.0.2.1` the primary IP where GitLab @@ -282,14 +276,13 @@ world. Custom domains are supported, but no TLS. ### Custom domains with TLS support -> **Requirements:** -> - [Wildcard DNS setup](#dns-configuration) -> - Wildcard TLS certificate -> - Secondary IP -> -> --- -> -> URL scheme: `https://page.example.io` and `https://domain.com` +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- Wildcard TLS certificate +- Secondary IP + +URL scheme: `https://page.example.io` and `https://domain.com` In that case, the pages daemon is running, Nginx still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -297,33 +290,33 @@ world. Custom domains and TLS are supported. 1. Install the Pages daemon: - ``` - cd /home/git - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git - cd gitlab-pages - sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) - sudo -u git -H make - ``` + ``` + cd /home/git + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git + cd gitlab-pages + sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) + sudo -u git -H make + ``` 1. Edit `gitlab.yml` to look like the example below. You need to change the `host` to the FQDN under which GitLab Pages will be served. Set `external_http` and `external_https` to the secondary IP on which the pages daemon will listen for connections: - ```yaml - ## GitLab Pages - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - # path: shared/pages + ```yaml + ## GitLab Pages + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + # path: shared/pages - host: example.io - port: 443 - https: true + host: example.io + port: 443 + https: true - external_http: 192.0.2.2:80 - external_https: 192.0.2.2:443 - ``` + external_http: 192.0.2.2:80 + external_https: 192.0.2.2:443 + ``` 1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in order to enable the pages daemon. In `gitlab_pages_options` the @@ -332,17 +325,17 @@ world. Custom domains and TLS are supported. The `-root-cert` and `-root-key` settings are the wildcard TLS certificates of the `example.io` domain: - ``` - gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80 -listen-https 192.0.2.2:443 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key - ``` + ``` + gitlab_pages_enabled=true + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 192.0.2.2:80 -listen-https 192.0.2.2:443 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key + ``` 1. Copy the `gitlab-pages-ssl` Nginx configuration file: - ```bash - sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf - sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf - ``` + ```bash + sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf + sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf + ``` 1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace `0.0.0.0` with `192.0.2.1`, where `192.0.2.1` the primary IP where GitLab @@ -359,9 +352,9 @@ are stored. If you wish to store them in another location you must set it up in `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['pages_path'] = "/mnt/storage/pages" - ``` + ```ruby + gitlab_rails['pages_path'] = "/mnt/storage/pages" + ``` 1. [Reconfigure GitLab][reconfigure] @@ -414,10 +407,10 @@ Pages access control is disabled by default. To enable it: 1. Modify your `config/gitlab.yml` file: - ```yaml - pages: - access_control: true - ``` + ```yaml + pages: + access_control: true + ``` 1. [Restart GitLab][restart]. 1. Create a new [system OAuth application](../../integration/oauth_provider.md#adding-an-application-through-the-profile). @@ -426,12 +419,12 @@ Pages access control is disabled by default. To enable it: application, but it does need the "api" scope. 1. Start the Pages daemon with the following additional arguments: - ```shell - -auth-client-secret <OAuth code generated by GitLab> \ - -auth-redirect-uri http://projects.example.io/auth \ - -auth-secret <40 random hex characters> \ - -auth-server <URL of the GitLab instance> - ``` + ```shell + -auth-client-secret <OAuth code generated by GitLab> \ + -auth-redirect-uri http://projects.example.io/auth \ + -auth-secret <40 random hex characters> \ + -auth-server <URL of the GitLab instance> + ``` 1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control-core-only). @@ -444,12 +437,12 @@ are stored. If you wish to store them in another location you must set it up in `gitlab.yml` under the `pages` section: - ```yaml - pages: - enabled: true - # The location where pages are stored (default: shared/pages). - path: /mnt/storage/pages - ``` + ```yaml + pages: + enabled: true + # The location where pages are stored (default: shared/pages). + path: /mnt/storage/pages + ``` 1. [Restart GitLab][restart] diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index 4302667caf5..92a4d56ca63 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -39,7 +39,7 @@ Follow the steps below to set up a custom hook: 1. Inside the `plugins` directory, create a file with a name of your choice, without spaces or special characters. -1. Make the hook file executable and make sure it's owned by the git user. +1. Make the hook file executable and make sure it's owned by the Git user. 1. Write the code to make the plugin function as expected. That can be in any language, and ensure the 'shebang' at the top properly reflects the language type. For example, if the script is in Ruby the shebang will @@ -52,7 +52,37 @@ as appropriate. The plugins file list is updated for each event, there is no need to restart GitLab to apply a new plugin. If a plugin executes with non-zero exit code or GitLab fails to execute it, a -message will be logged to `plugin.log`. +message will be logged to: + +- `gitlab-rails/plugin.log` in an Omnibus installation. +- `log/plugin.log` in a source installation. + +## Creating plugins + +Below is an example that will only response on the event `project_create` and +will inform the admins from the GitLab instance that a new project has been created. + +```ruby +# By using the embedded ruby version we eliminate the possibility that our chosen language +# would be unavailable from +#!/opt/gitlab/embedded/bin/ruby +require 'json' +require 'mail' + +# The incoming variables are in JSON format so we need to parse it first. +ARGS = JSON.parse(STDIN.read) + +# We only want to trigger this plugin on the event project_create +return unless ARGS['event_name'] == 'project_create' + +# We will inform our admins of our gitlab instance that a new project is created +Mail.deliver do + from 'info@gitlab_instance.com' + to 'admin@gitlab_instance.com' + subject "new project " + ARGS['name'] + body ARGS['owner_name'] + 'created project ' + ARGS['name'] +end +``` ## Validation @@ -82,4 +112,4 @@ Validating plugins from /plugins directory [system hooks]: ../system_hooks/system_hooks.md [webhooks]: ../user/project/integrations/webhooks.md -[highly available]: ./high_availability/README.md
\ No newline at end of file +[highly available]: ./high_availability/README.md diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 78b2751da13..716a4259a64 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -65,8 +65,8 @@ To configure the pseudonymizer, you need to: ```yaml pseudonymizer: - manifest: config/pseudonymizer.yml - upload: + manifest: config/pseudonymizer.yml + upload: remote_directory: 'gitlab-elt' # bucket name connection: provider: AWS diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 435aed8c413..691d34ab7fa 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -2,7 +2,7 @@ ## Git housekeeping -There are few tasks you can run to schedule a git housekeeping to start at the +There are few tasks you can run to schedule a git housekeeping to start at the next repository sync in a **Secondary node**: ### Incremental Repack @@ -23,7 +23,7 @@ sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_EN ### Full Repack -This is equivalent of running `git repack -d -A --pack-kept-objects` on a +This is equivalent of running `git repack -d -A --pack-kept-objects` on a _bare_ repository which will optionally, write a reachability bitmap index when this is enabled in GitLab. diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 91fc0133d56..e880d76e756 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -28,6 +28,31 @@ limit by passing a number to the check task: rake gitlab:ldap:check[50] ``` +## Run a Group Sync + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.3. + +The following task will run a [group sync](../auth/ldap-ee.md#group-sync) immediately. This is valuable +when you'd like to update all configured group memberships against LDAP without +waiting for the next scheduled group sync to be run. + +NOTE: **NOTE:** +If you'd like to change the frequency at which a group sync is performed, +[adjust the cron schedule](../auth/ldap-ee.md#adjusting-ldap-group-sync-schedule) +instead. + +**Omnibus Installation** + +``` +sudo gitlab-rake gitlab:ldap:group_sync +``` + +**Source Installation** + +```bash +bundle exec rake gitlab:ldap:group_sync +``` + ## Rename a provider If you change the LDAP server ID in `gitlab.yml` or `gitlab.rb` you will need diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 2b31233d429..8d0b5b42515 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -251,3 +251,22 @@ sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:*] # to clear a lease for repository garbage collection in a specific project: (id=4) sudo gitlab-rake gitlab:exclusive_lease:clear[project_housekeeping:4] ``` + +## Display status of database migrations + +To check the status of migrations, you can use the following rake task: + +```bash +sudo gitlab-rake db:migrate:status +``` + +This will output a table with a `Status` of `up` or `down` for +each Migration ID. + +```bash +database: gitlabhq_production + + Status Migration ID Migration Name +-------------------------------------------------- + up migration_id migration_name +```
\ No newline at end of file diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 0599e12b913..138db4dfbb1 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -2,14 +2,14 @@ >**Note:** > -> - [Introduced][ce-3050] in GitLab 8.9. -> - Importing will not be possible if the import instance version is lower -> than that of the exporter. -> - For existing installations, the project import option has to be enabled in -> application settings (`/admin/application_settings`) under 'Import sources'. -> - The exports are stored in a temporary [shared directory][tmp] and are deleted -> every 24 hours by a specific worker. -> - ImportExport can use object storage automatically starting from GitLab 11.3 +> - [Introduced][ce-3050] in GitLab 8.9. +> - Importing will not be possible if the import instance version is lower +> than that of the exporter. +> - For existing installations, the project import option has to be enabled in +> application settings (`/admin/application_settings`) under 'Import sources'. +> - The exports are stored in a temporary [shared directory][tmp] and are deleted +> every 24 hours by a specific worker. +> - ImportExport can use object storage automatically starting from GitLab 11.3 The GitLab Import/Export version can be checked by using: diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 42a1a1c2e60..1198f3414c5 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,17 +1,17 @@ # Repository Storage Rake Tasks -This is a collection of rake tasks you can use to help you list and migrate -existing projects and attachments associated with it from Legacy storage to +This is a collection of rake tasks you can use to help you list and migrate +existing projects and attachments associated with it from Legacy storage to the new Hashed storage type. You can read more about the storage types [here][storage-types]. ## Migrate existing projects to Hashed storage -Before migrating your existing projects, you should +Before migrating your existing projects, you should [enable hashed storage][storage-migration] for the new projects as well. -This task will schedule all your existing projects and attachments associated with it to be migrated to the +This task will schedule all your existing projects and attachments associated with it to be migrated to the **Hashed** storage type: **Omnibus Installation** @@ -30,15 +30,15 @@ They both also accept a range as environment variable: ```bash # to migrate any non migrated project from ID 20 to 50. -export ID_FROM=20 +export ID_FROM=20 export ID_TO=50 ``` You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -There is a specific Queue you can watch to see how long it will take to finish: +There is a specific Queue you can watch to see how long it will take to finish: `hashed_storage:hashed_storage_project_migrate` -After it reaches zero, you can confirm every project has been migrated by running the commands bellow. +After it reaches zero, you can confirm every project has been migrated by running the commands bellow. If you find it necessary, you can run this migration script again to schedule missing projects. Any error or warning will be logged in Sidekiq's log file. @@ -55,7 +55,7 @@ If you need to rollback the storage migration for any reason, you can follow the NOTE: **Note:** Hashed Storage will be required in future version of GitLab. -To prevent new projects from being created in the Hashed storage, +To prevent new projects from being created in the Hashed storage, you need to undo the [enable hashed storage][storage-migration] changes. This task will schedule all your existing projects and associated attachments to be rolled back to the @@ -77,15 +77,14 @@ Both commands accept a range as environment variable: ```bash # to rollback any migrated project from ID 20 to 50. -export ID_FROM=20 +export ID_FROM=20 export ID_TO=50 ``` You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. - -After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. +After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. Any error or warning will be logged in Sidekiq's log file. @@ -106,8 +105,6 @@ sudo gitlab-rake gitlab:storage:legacy_projects sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production ``` ------- - To list projects using **Legacy** storage: **Omnibus Installation** @@ -139,8 +136,6 @@ sudo gitlab-rake gitlab:storage:hashed_projects sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production ``` ------- - To list projects using **Hashed** storage: **Omnibus Installation** @@ -171,8 +166,6 @@ sudo gitlab-rake gitlab:storage:legacy_attachments sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production ``` ------- - To list project attachments using **Legacy** storage: **Omnibus Installation** @@ -203,8 +196,6 @@ sudo gitlab-rake gitlab:storage:hashed_attachments sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production ``` ------- - To list project attachments using **Hashed** storage: **Omnibus Installation** diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index fd8ea8d3162..86e8b763f51 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -103,3 +103,13 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, S sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" ``` + +## Migrate legacy uploads out of deprecated paths + +> Introduced in GitLab 12.3. + +To migrate all uploads created by legacy uploaders, run: + +```shell +bundle exec rake gitlab:uploads:legacy:migrate +``` diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 54a423b9571..ae5ccfb9e37 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -4,16 +4,16 @@ You need `exiftool` installed on your system. If you installed GitLab: -- Using the Omnibus package, you're all set. -- From source, make sure `exiftool` is installed: +- Using the Omnibus package, you're all set. +- From source, make sure `exiftool` is installed: - ```sh - # Debian/Ubuntu - sudo apt-get install libimage-exiftool-perl + ```sh + # Debian/Ubuntu + sudo apt-get install libimage-exiftool-perl - # RHEL/CentOS - sudo yum install perl-Image-ExifTool - ``` + # RHEL/CentOS + sudo yum install perl-Image-ExifTool + ``` ## Remove EXIF data from existing uploads diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 406f7e8a034..56cd23b2eb8 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -18,7 +18,7 @@ The instructions make the assumption that you will be using the email address `i sudo apt-get install postfix ``` - When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches gitlab.example.com`. + When asked about the environment, select 'Internet Site'. When asked to confirm the hostname, make sure it matches `gitlab.example.com`. 1. Install the `mailutils` package. @@ -331,7 +331,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo a logout ``` -## Done! +## Done If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email] guide to configure GitLab. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 7cf8f20a9dc..ab911c1cf0e 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -3,7 +3,7 @@ > [Introduced][ce-3232] in GitLab 8.7. It is OFF by default because it still causes too many false alarms. -Git has a built-in mechanism, [git fsck][git-fsck], to verify the +Git has a built-in mechanism, [`git fsck`][git-fsck], to verify the integrity of all data committed to a repository. GitLab administrators can trigger such a check for a project via the project page under the admin panel. The checks run asynchronously so it may take a few minutes @@ -33,8 +33,8 @@ in `repocheck.log`: - in the [admin panel](logs.md#repochecklog) - or on disk, see: - - `/var/log/gitlab/gitlab-rails` for Omnibus installations - - `/home/git/gitlab/log` for installations from source + - `/var/log/gitlab/gitlab-rails` for Omnibus installations + - `/home/git/gitlab/log` for installations from source If for some reason the periodic repository check caused a lot of false alarms you can choose to clear *all* repository check states by diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 3de860f9240..b1a870210a8 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -55,14 +55,12 @@ storage2: > don't specify the full repository path but the parent path), but not for source > installations. ---- - Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs` and `cephfs` +the example below, we add two more mountpoints that are named `nfs_1` and `nfs_2` respectively. -NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** @@ -75,10 +73,10 @@ NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS storages: # You must have at least a 'default' storage path. default: path: /home/git/repositories - nfs: - path: /mnt/nfs/repositories - cephfs: - path: /mnt/cephfs/repositories + nfs_1: + path: /mnt/nfs1/repositories + nfs_2: + path: /mnt/nfs2/repositories ``` 1. [Restart GitLab][restart-gitlab] for the changes to take effect. @@ -90,8 +88,6 @@ are upgrading from a version prior to 8.10, make sure to add the configuration as described in the step above. After you make the changes and confirm they are working, you can remove the `repos_path` line. ---- - **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the @@ -100,8 +96,8 @@ working, you can remove the `repos_path` line. ```ruby git_data_dirs({ "default" => { "path" => "/var/opt/gitlab/git-data" }, - "nfs" => { "path" => "/mnt/nfs/git-data" }, - "cephfs" => { "path" => "/mnt/cephfs/git-data" } + "nfs_1" => { "path" => "/mnt/nfs1/git-data" }, + "nfs_2" => { "path" => "/mnt/nfs2/git-data" } }) ``` diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 9dea6074a3f..3ee8673b297 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -82,19 +82,17 @@ by another folder with the next 2 characters. They are both stored in a special > [Introduced](https://gitlab.com/gitlab-org/gitaly/issues/1606) in GitLab 12.1. -Forks of public projects are deduplicated by creating a third repository, the object pool, containing the objects from the source project. Using `objects/info/alternates`, the source project and forks use the object pool for shared objects. Objects are moved from the source project to the object pool when housekeeping is run on the source project. +Forks of public projects are deduplicated by creating a third repository, the +object pool, containing the objects from the source project. Using +`objects/info/alternates`, the source project and forks use the object pool for +shared objects. Objects are moved from the source project to the object pool +when housekeeping is run on the source project. ```ruby # object pool paths "@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" ``` -Object pools can be disabled using the `object_pools` feature flag, and can be -disabled for individual projects by executing -`Feature.disable(:object_pools, Project.find(<id>))`. Disabling object pools -will not change existing deduplicated forks, but will prevent new forks from -being deduplicated. - DANGER: **Danger:** Do not run `git prune` or `git gc` in pool repositories! This can cause data loss in "real" repositories that depend on the pool in @@ -105,7 +103,7 @@ question. To start a migration, enable Hashed Storage for new projects: 1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -2. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. +1. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. Check if the change breaks any existing integration you may have that either runs on the same machine as your repositories are located, or may login to that machine @@ -118,7 +116,7 @@ If you do have any existing integration, you may want to do a small rollout firs to validate. You can do so by specifying a range with the operation. This is an example of how to limit the rollout to Project IDs 50 to 100, running in -an Omnibus Gitlab installation: +an Omnibus GitLab installation: ```bash sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 @@ -133,13 +131,13 @@ Similar to the migration, to disable Hashed Storage for new projects: 1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -2. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. +1. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. To schedule a complete rollback, see the [rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. The rollback task also supports specifying a range of Project IDs. Here is an example -of limiting the rollout to Project IDs 50 to 100, in an Omnibus Gitlab installation: +of limiting the rollout to Project IDs 50 to 100, in an Omnibus GitLab installation: ```bash sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 @@ -185,7 +183,7 @@ CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in ##### LFS Objects -LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following git own implementation: +LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following Git own implementation: ```ruby "shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md new file mode 100644 index 00000000000..b2e3bf8487b --- /dev/null +++ b/doc/administration/smime_signing_email.md @@ -0,0 +1,49 @@ +# Signing outgoing email with S/MIME + +Notification emails sent by Gitlab can be signed with S/MIME for improved +security. + +> **Note:** +Please be aware that S/MIME certificates and TLS/SSL certificates are not the +same and are used for different purposes: TLS creates a secure channel, whereas +S/MIME signs and/or encrypts the message itself + +## Enable S/MIME signing + +This setting must be explicitly enabled and a single pair of key and certificate +files must be provided in `gitlab.rb` or `gitlab.yml` if you are using Omnibus +GitLab or installed GitLab from source respectively: + +```yaml +email_smime: + enabled: true + key_file: /etc/pki/smime/private/gitlab.key + cert_file: /etc/pki/smime/certs/gitlab.crt +``` + +- Both files must be provided PEM-encoded. +- The key file must be unencrypted so that Gitlab can read it without user + intervention. + +NOTE: **Note:** Be mindful of the access levels for your private keys and visibility to +third parties. + +### How to convert S/MIME PKCS#12 / PFX format to PEM encoding + +Typically S/MIME certificates are handled in binary PKCS#12 format (`.pfx` or `.p12` +extensions), which contain the following in a single encrypted file: + +- Server certificate +- Intermediate certificates (if any) +- Private key + +In order to export the required files in PEM encoding from the PKCS#12 file, +the `openssl` command can be used: + +```bash +#-- Extract private key in PEM encoding (no password, unencrypted) +$ openssl pkcs12 -in gitlab.p12 -nocerts -nodes -out gitlab.key + +#-- Extract certificates in PEM encoding (full certs chain including CA) +$ openssl pkcs12 -in gitlab.p12 -nokeys -out gitlab.crt +``` diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 8f7280d5128..604dff5983d 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -10,43 +10,43 @@ an SMTP server, but you're not seeing mail delivered. Here's how to check the se 1. Run a Rails console: - ```sh - sudo gitlab-rails console production - ``` + ```sh + sudo gitlab-rails console production + ``` - or for source installs: + or for source installs: - ```sh - bundle exec rails console production - ``` + ```sh + bundle exec rails console production + ``` 1. Look at the ActionMailer `delivery_method` to make sure it matches what you intended. If you configured SMTP, it should say `:smtp`. If you're using Sendmail, it should say `:sendmail`: - ```ruby - irb(main):001:0> ActionMailer::Base.delivery_method - => :smtp - ``` + ```ruby + irb(main):001:0> ActionMailer::Base.delivery_method + => :smtp + ``` 1. If you're using SMTP, check the mail settings: - ```ruby - irb(main):002:0> ActionMailer::Base.smtp_settings - => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true}``` - ``` + ```ruby + irb(main):002:0> ActionMailer::Base.smtp_settings + => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true}``` + ``` - In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail - logs (e.g. `/var/log/mail.log`) for more details. + In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail + logs (e.g. `/var/log/mail.log`) for more details. -1. Send a test message via the console. +1. Send a test message via the console. - ```ruby - irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now - ``` + ```ruby + irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now + ``` - If you do not receive an e-mail and/or see an error message, then check - your mail server settings. + If you do not receive an e-mail and/or see an error message, then check + your mail server settings. ## Advanced Issues @@ -103,37 +103,37 @@ downtime. Otherwise skip to the next section. 1. Run `sudo gdb -p <PID>` to attach to the unicorn process. 1. In the gdb window, type: - ``` - call (void) rb_backtrace() - ``` + ``` + call (void) rb_backtrace() + ``` 1. This forces the process to generate a Ruby backtrace. Check `/var/log/gitlab/unicorn/unicorn_stderr.log` for the backtace. For example, you may see: - ```ruby - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' - ``` + ```ruby + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' + ``` 1. To see the current threads, run: - ``` - thread apply all bt - ``` + ``` + thread apply all bt + ``` 1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: - ``` - detach - exit - ``` + ``` + detach + exit + ``` Note that if the unicorn process terminates before you are able to run these commands, gdb will report an error. To buy more time, you can always raise the @@ -162,21 +162,21 @@ separate Rails process to debug the issue: 1. Create a Personal Access Token for your user (Profile Settings -> Access Tokens). 1. Bring up the GitLab Rails console. For omnibus users, run: - ``` - sudo gitlab-rails console - ``` + ``` + sudo gitlab-rails console + ``` 1. At the Rails console, run: - ```ruby - [1] pry(main)> app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' - ``` + ```ruby + [1] pry(main)> app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' + ``` - For example: + For example: - ```ruby - [1] pry(main)> app.get 'https://gitlab.com/gitlab-org/gitlab-ce/issues/1?private_token=123456' - ``` + ```ruby + [1] pry(main)> app.get 'https://gitlab.com/gitlab-org/gitlab-ce/issues/1?private_token=123456' + ``` 1. In a new window, run `top`. It should show this ruby process using 100% CPU. Write down the PID. 1. Follow step 2 from the previous section on using gdb. @@ -209,7 +209,7 @@ ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -s 10 The output in `/tmp/unicorn.txt` may help diagnose the root cause. -# More information +## More information - [Debugging Stuck Ruby Processes](https://blog.newrelic.com/2013/04/29/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9/) - [Cheatsheet of using gdb and ruby processes](gdb-stuck-ruby.txt) diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md new file mode 100644 index 00000000000..ab3b25f0e97 --- /dev/null +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -0,0 +1,27 @@ +--- +type: reference +--- + +# Diagnostics tools + +These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. +They are listed here for transparency, and they may be useful for users with experience +with troubleshooting GitLab. If you are currently having an issue with GitLab, you +may want to check your [support options](https://about.gitlab.com/support/) first, +before attempting to use these tools. + +## gitlabsos + +The [gitlabsos](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) utility +provides a unified method of gathering info and logs from GitLab and the system it's +running on. + +## strace-parser + +[strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze +and summarize raw strace data. + +## Pritaly + +[Pritaly](https://gitlab.com/wchandler/pritaly) takes Gitaly logs and colorizes output +or converts the logs to JSON. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md new file mode 100644 index 00000000000..c4a7ba01fae --- /dev/null +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -0,0 +1,345 @@ +# Troubleshooting ElasticSearch + +Troubleshooting ElasticSearch requires: + +- Knowledge of common terms. +- Establishing within which category the problem fits. + +## Common terminology + +- **Lucene**: A full-text search library written in Java. +- **Near Realtime (NRT)**: Refers to the slight latency from the time to index a + document to the time when it becomes searchable. +- **Cluster**: A collection of one or more nodes that work together to hold all + the data, providing indexing and search capabilities. +- **Node**: A single server that works as part of a cluster. +- **Index**: A collection of documents that have somewhat similar characteristics. +- **Document**: A basic unit of information that can be indexed. +- **Shards**: Fully-functional and independent subdivisions of indices. Each shard is actually + a Lucene index. +- **Replicas**: Failover mechanisms that duplicate indices. + +## Troubleshooting workflows + +The type of problem will determine what steps to take. The possible troubleshooting workflows are for: + +- Search results. +- Indexing. +- Integration. +- Performance. + +### Search Results workflow + +The following workflow is for ElasticSearch search results issues: + +```mermaid +graph TD; + B --> |No| B1 + B --> |Yes| B4 + B1 --> B2 + B2 --> B3 + B4 --> B5 + B5 --> |Yes| B6 + B5 --> |No| B7 + B7 --> B8 + B{Is GitLab using<br>ElasticSearch for<br>searching?} + B1[Check Admin Area > Integrations<br>to ensure the settings are correct] + B2[Perform a search via<br>the rails console] + B3[If all settings are correct<br>and it still doesn't show ElasticSearch<br>doing the searches, escalate<br>to GitLab support.] + B4[Perform<br>the same search via the<br>ElasticSearch API] + B5{Are the results<br>the same?} + B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.] + B7[Check the index status of the project<br>containing the missing search<br>results.] + B8(Indexing Troubleshooting) +``` + +### Indexing workflow + +The following workflow is for ElasticSearch indexing issues: + +```mermaid +graph TD; + C --> |Yes| C1 + C1 --> |Yes| C2 + C1 --> |No| C3 + C3 --> |Yes| C4 + C3 --> |No| C5 + C --> |No| C6 + C6 --> |No| C10 + C7 --> |GitLab| C8 + C7 --> |ElasticSearch| C9 + C6 --> |Yes| C7 + C10 --> |No| C12 + C10 --> |Yes| C11 + C12 --> |Yes| C13 + C12 --> |No| C14 + C14 --> |Yes| C15 + C14 --> |No| C16 + C{Is the problem with<br>creating an empty<br>index?} + C1{Does the gitlab-production<br>index exist on the<br>ElasticSearch instance?} + C2(Try to manually<br>delete the index on the<br>ElasticSearch instance and<br>retry creating an empty index.) + C3{Can indices be made<br>manually on the ElasticSearch<br>instance?} + C4(Retry the creation of an empty index) + C5(It is best to speak with an<br>ElasticSearch admin concerning the<br>instance's inability to create indices.) + C6{Is the indexer presenting<br>errors during indexing?} + C7{Is the error a GitLab<br>error or an ElasticSearch<br>error?} + C8[Escalate to<br>GitLab support] + C9[You will want<br>to speak with an<br>ElasticSearch admin.] + C10{Does the index status<br>show 100%?} + C11[Escalate to<br>GitLab support] + C12{Does re-indexing the project<br> present any GitLab errors?} + C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.] + C14{Does re-indexing the project<br>present errors on the <br>ElasticSearch instance?} + C15[It would be best<br>to speak with an<br>ElasticSearch admin.] + C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] +``` + +### Integration workflow + +The following workflow is for ElasticSearch integration issues: + +```mermaid +graph TD; + D --> |No| D1 + D --> |Yes| D2 + D2 --> |No| D3 + D2 --> |Yes| D4 + D4 --> |No| D5 + D4 --> |Yes| D6 + D{Is the error concerning<br>the beta indexer?} + D1[It would be best<br>to speak with an<br>ElasticSearch admin.] + D2{Is the ICU development<br>package installed?} + D3>This package is required.<br>Install the package<br>and retry.] + D4{Is the error stemming<br>from the indexer?} + D5[This would indicate an OS level<br> issue. It would be best to<br>contact your sysadmin.] + D6[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] +``` + +### Performance workflow + +The following workflow is for ElasticSearch performance issues: + +```mermaid +graph TD; + F --> |Yes| F1 + F --> |No| F2 + F2 --> |No| F3 + F2 --> |Yes| F4 + F4 --> F5 + F5 --> |No| F6 + F5 --> |Yes| F7 + F{Is the ElasticSearch instance<br>running on the same server<br>as the GitLab instance?} + F1(This is not advised and will cause issues.<br>We recommend moving the ElasticSearch<br>instance to a different server.) + F2{Does the ElasticSearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?} + F3(According to ElasticSearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.) + F4(Obtain the <br>cluster health information) + F5(Does it show the<br>status as green?) + F6(We recommend you speak with<br>an ElasticSearch admin<br>about implementing sharding.) + F7(Escalate to<br>GitLab support.) +``` + +## Troubleshooting walkthrough + +Most ElasticSearch troubleshooting can be broken down into 4 categories: + +- [Troubleshooting search results](#troubleshooting-search-results) +- [Troubleshooting indexing](#troubleshooting-indexing) +- [Troubleshooting integration](#troubleshooting-integration) +- [Troubleshooting performance](#troubleshooting-performance) + +Generally speaking, if it does not fall into those four categories, it is either: + +- Something GitLab support needs to look into. +- Not a true ElasticSearch issue. + +Exercise caution. Issues that appear to be ElasticSearch problems can be OS-level issues. + +### Troubleshooting search results + +Troubleshooting search result issues is rather straight forward on ElasticSearch. + +The first step is to confirm GitLab is using ElasticSearch for the search function. +To do this: + +1. Confirm the integration is enabled in **Admin Area > Settings > Integrations**. +1. Confirm searches utilize ElasticSearch by accessing the rails console + (`sudo gitlab-rails console`) and running the following commands: + + ```rails + u = User.find_by_email('email_of_user_doing_search') + s = SearchService.new(u, {:search => 'search_term'}) + pp s.search_objects.class.name + ``` + +The ouput from the last command is the key here. If it shows: + +- `ActiveRecord::Relation`, **it is not** using ElasticSearch. +- `Kaminari::PaginatableArray`, **it is** using ElasticSearch. + +| Not using ElasticSearch | Using ElasticSearch | +|--------------------------|------------------------------| +| `ActiveRecord::Relation` | `Kaminari::PaginatableArray` | + +If all the settings look correct and it is still not using ElasticSearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. + +Moving past that, it is best to attempt the same search using the [ElasticSearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab. + +If the results: + +- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem + with the ElasticSearch filters we are using. This can be complicated, so it is best to + escalate to GitLab support to check these and guide you on the potential on whether or + not a feature request is needed. +- Do not match up, this indicates a problem with the documents generated from the + project. It is best to re-index that project and proceed with + [Troubleshooting indexing](#troubleshooting-indexing). + +### Troubleshooting indexing + +Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab +support or your ElasticSearch admin. + +The best place to start is to determine if the issue is with creating an empty index. +If it is, check on the ElasticSearch side to determine if the `gitlab-production` (the +name for the GitLab index) exists. If it exists, manually delete it on the ElasticSearch +side and attempt to recreate it from the +[`create_empty_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +rake task. + +If you still encounter issues, try creating an index manually on the ElasticSearch +instance. The details of the index aren't important here, as we want to test if indices +can be made. If the indices: + +- Cannot be made, speak with your ElasticSearch admin. +- Can be made, Escalate this to GitLab support. + +If the issue is not with creating an empty index, the next step is to check for errors +during the indexing of projects. If errors do occur, they will either stem from the indexing: + +- On the GitLab side. You need to rectify those. If they are not + something you are familiar with, contact GitLab support for guidance. +- Within the ElasticSearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your ElasticSearch admin. + +If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following rake tasks: + +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows specific projects that are not indexed) + +If: + +- Everything is showing at 100%, escalate to GitLab support. This could be a potential + bug/issue. +- You do see something not at 100%, attempt to reindex that project. To do this, + run `sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=<project ID> ID_TO=<project ID>`. + +If reindexing the project shows: + +- Errors on the GitLab side, escalate those to GitLab support. +- ElasticSearch errors or doesn't present any errors at all, reach out to your + ElasticSearch admin to check the instance. + +### Troubleshooting integration + +Troubleshooting integration tends to be pretty straight forward, as there really isn't +much to "integrate" here. + +If the issue is: + +- Not concerning the beta indexer, it is almost always an + ElasticSearch-side issue. This means you should reach out to your ElasticSearch admin + regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach + out to GitLab support. +- With the beta indexer, check if the ICU development package is installed. + This is a required package so make sure you install it. + +Beyond that, you will want to review the error. If it is: + +- Specifically from the indexer, this could be a bug/issue and should be escalated to + GitLab support. +- An OS issue, you will want to reach out to your systems administrator. + +### Troubleshooting performance + +Troubleshooting performance can be difficult on ElasticSearch. There is a ton of tuning +that *can* be done, but the majority of this falls on shoulders of a skilled +ElasticSearch administrator. + +Generally speaking, ensure: + +* The ElasticSearch server **is not** running on the same node as GitLab. +* The ElasticSearch server have enough RAM and CPU cores. +* That sharding **is** being used. + +Going into some more detail here, if ElasticSearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, ElasticSearch, which requires ample resources, should be running on its own server (maybe coupled with logstash and kibana). + +When it comes to ElasticSearch, RAM is the key resource. ElasticSearch themselves recommend: + +- **At least** 8 GB of RAM for a non-production instance. +- **At least** 16 GB of RAM for a production instance. +- Ideally, 64 GB of RAM. + +For CPU, ElasticSearch recommends at least 2 CPU cores, but ElasticSearch states common +setups use up to 8 cores. For more details on server specs, check out +[ElasticSearch's hardware guide](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). + +Beyond the obvious, sharding comes into play. Sharding is a core part of ElasticSearch. +It allows for horizontal scaling of indices, which is helpful when you are dealing with +a large amount of data. + +With the way GitLab does indexing, there is a **huge** amount of documents being +indexed. By utilizing sharding, you can speed up ElasticSearch's ability to locate +data, since each shard is a Lucene index. + +If you are not using sharding, you are likely to hit issues when you start using +ElasticSearch in a production environment. + +Keep in mind that an index with only one shard has **no scale factor** and will +likely encounter issues when called upon with some frequency. + +If you need to know how many shards, read +[ElasticSearch's documentation on capacity planning](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/capacity-planning.html), +as the answer is not straight forward. + +The easiest way to determine if sharding is in use is to check the output of the +[ElasticSearch Health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): + +- Red means the cluster is down. +- Yellow means it is up with no sharding/replication. +- Green means it is healthy (up, sharding, replicating). + +For production use, it should always be green. + +Beyond these steps, you get into some of the more complicated things to check, +such as merges and caching. These can get complicated and it takes some time to +learn them, so it is best to escalate/pair with an ElasticSearch expert if you need to +dig further into these. + +Feel free to reach out to GitLab support, but this is likely to be something a skilled +ElasticSearch admin has more experience with. + +## Common issues + +All common issues [should be documented](../../integration/elasticsearch.md#troubleshooting). If not, +feel free to update that page with issues you encounter and solutions. + +## Replication + +Setting up ElasticSearch isn't too bad, but it can be a bit finnicky and time consuming. + +The eastiest method is to spin up a docker container with the required version and +bind ports 9200/9300 so it can be used. + +The following is an example of running a docker container of ElasticSearch v7.2.0: + +```bash +docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 +docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.2.0 +``` + +From here, you can: + +- Grab the IP of the docker container (use `docker inspect <container_id>`) +- Use `<IP.add.re.ss:9200>` to communicate with it. + +This is a quick method to test out ElasticSearch, but by no means is this a +production solution. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md new file mode 100644 index 00000000000..260af333e8e --- /dev/null +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -0,0 +1,261 @@ +--- +type: reference +--- + +# Kubernetes, GitLab and You + +This is a list of useful information regarding Kubernetes that the GitLab Support +Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone +can make use of the Support team's collected knowledge + +CAUTION: **Caution:** +These commands **can alter or break** your Kubernetes components so use these at your own risk. + +If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how +to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) +and they will assist you with any issues you are having. + +## Generic kubernetes commands + +- How to authorize to your GCP project (can be especially useful if you have projects + under different GCP accounts): + + ```bash + gcloud auth login + ``` + +- How to access Kubernetes dashboard: + + ```bash + # for minikube: + minikube dashboard —url + # for non-local installations if access via Kubectl is configured: + kubectl proxy + ``` + +- How to ssh to a Kubernetes node and enter the container as root + <https://github.com/kubernetes/kubernetes/issues/30656>: + + - For GCP, you may find the node name and run `gcloud compute ssh node-name`. + - List containers using `docker ps`. + - Enter container using `docker exec --user root -ti container-id bash`. + +- How to copy a file from local machine to a pod: + + ```bash + kubectl cp file-name pod-name:./destination-path + ``` + +- What to do with pods in `CrashLoopBackoff` status: + + - Check logs via Kubernetes dashboard. + - Check logs via Kubectl: + + ```bash + kubectl logs <unicorn pod> -c dependencies + ``` + +- How to tail all Kubernetes cluster events in real time: + + ```bash + kubectl get events -w --all-namespaces + ``` + +- How to get logs of the previously terminated pod instance: + + ```bash + kubectl logs <pod-name> --previous + ``` + + NOTE: **Note:** + No logs are kept in the containers/pods themselves, everything is written to stdout. + This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) + for details. + +## GitLab-specific kubernetes information + +- Minimal config that can be used to test a Kubernetes helm chart can be found + [here](https://gitlab.com/charts/gitlab/issues/620). + +- Tailing logs of a separate pod. An example for a unicorn pod: + + ```bash + kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn + ``` + +- Tail and follow all pods that share a label (in this case, `unicorn`): + + ```bash + # all containers in the unicorn pods + kubectl logs -f -l app=unicorn --all-containers=true --max-log-requests=50 + + # only the unicorn containers in all unicorn pods + kubectl logs -f -l app=unicorn -c unicorn --max-log-requests=50 + ``` + +- One can stream logs from all containers at once, similar to the Omnibus + command `gitlab-ctl tail`: + + ```bash + kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100 + ``` + +- Check all events in the `gitlab` namespace (the namespace name can be different if you + specified a different one when deploying the helm chart): + + ```bash + kubectl get events -w --namespace=gitlab + ``` + +- Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner + pod. You may enter it and run commands inside or run them from the outside: + + ```bash + # find the pod + kubectl get pods | grep task-runner + + # enter it + kubectl exec -it <task-runner-pod-name> bash + + # open rails console + # rails console can be also called from other GitLab pods + /srv/gitlab/bin/rails console + + # source-style commands should also work + /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production + + # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart + /usr/local/bin/gitlab-rake gitlab:check + + # open console without entering pod + kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console + + # check the status of DB migrations + kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status + ``` + + You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. + +- Troubleshooting **Operations > Kubernetes** integration: + + - Check the output of `kubectl get events -w --all-namespaces`. + - Check the logs of pods within `gitlab-managed-apps` namespace. + - On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed + via Helm Chart, `kubernetes.log` can be found inside the sidekiq pod. + +- How to get your initial admin password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: + + ```bash + # find the name of the secret containing the password + kubectl get secrets | grep initial-root + # decode it + kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo + ``` + +- How to connect to a GitLab Postgres database: + + ```bash + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p + ``` + +- How to get info about Helm installation status: + + ```bash + helm status name-of-installation + ``` + +- How to update GitLab installed using Helm Chart: + + ```bash + helm repo upgrade + + # get current values and redirect them to yaml file (analogue of gitlab.rb values) + helm get values <release name> > gitlab.yaml + + # run upgrade itself + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + + After <https://canary.gitlab.com/charts/gitlab/issues/780> is fixed, it should + be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#updating-gitlab-using-the-helm-chart) + for upgrades. + +- How to apply changes to GitLab config: + + - Modify the `gitlab.yaml` file. + - Run the following command to apply changes: + + ```bash + helm upgrade <release name> <chart path> -f gitlab.yaml + ``` + +## Installation of minimal GitLab config via Minukube on macOS + +This section is based on [Developing for Kubernetes with Minikube](https://gitlab.com/charts/gitlab/blob/master/doc/minikube/index.md) +and [Helm](https://gitlab.com/charts/gitlab/blob/master/doc/helm/index.md). Refer +to those documents for details. + +- Install Kubectl via Homebrew: + + ```bash + brew install kubernetes-cli + ``` + +- Install Minikube via Homebrew: + + ```bash + brew cask install minikube + ``` + +- Start Minikube and configure it. If Minikube cannot start, try running `minikube delete && minikube start` + and repeat the steps: + + ```bash + minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work + minikube addons enable ingress + minikube addons enable kube-dns + ``` + +- Install Helm via Homebrew and initialize it: + + ```bash + brew install kubernetes-helm + helm init --service-account tiller + ``` + +- Copy the file <https://gitlab.com/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml> + to your workstation. + +- Find the IP address in the output of `minikube ip` and update the yaml file with + this IP address. + +- Install the GitLab Helm Chart: + + ```bash + helm repo add gitlab https://charts.gitlab.io + helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab + ``` + + If you want to modify some GitLab settings, you can use the above-mentioned config + as a base and create your own yaml file. + +- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. + The installation could take up to 20-30 minutes depending on the amount of resources + on your workstation. + +- When all the pods show either a `Running` or `Completed` status, get the GitLab password as + described in [Initial login](https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html#initial-login), + and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` + where `domain` is the value provided in the yaml file. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 7067958ecb4..9b016c64e29 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -169,3 +169,121 @@ The PostgreSQL wiki has details on the query you can run to see blocking queries. The query is different based on PostgreSQL version. See [Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for the query details. + +## Managing Sidekiq queues + +It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) +to perform a number of troubleshoting on Sidekiq. + +These are the administrative commands and it should only be used if currently +admin interface is not suitable due to scale of installation. + +All this commands should be run using `gitlab-rails console`. + +### View the queue size + +```ruby +Sidekiq::Queue.new("pipeline_processing:build_queue").size +``` + +### Enumerate all enqueued jobs + +```ruby +queue = Sidekiq::Queue.new("chaos:chaos_sleep") +queue.each do |job| + # job.klass # => 'MyWorker' + # job.args # => [1, 2, 3] + # job.jid # => jid + # job.queue # => chaos:chaos_sleep + # job["retry"] # => 3 + # job.item # => { + # "class"=>"Chaos::SleepWorker", + # "args"=>[1000], + # "retry"=>3, + # "queue"=>"chaos:chaos_sleep", + # "backtrace"=>true, + # "queue_namespace"=>"chaos", + # "jid"=>"39bc482b823cceaf07213523", + # "created_at"=>1566317076.266069, + # "correlation_id"=>"c323b832-a857-4858-b695-672de6f0e1af", + # "enqueued_at"=>1566317076.26761}, + # } + + # job.delete if job.jid == 'abcdef1234567890' +end +``` + +### Enumerate currently running jobs + +```ruby +workers = Sidekiq::Workers.new +workers.each do |process_id, thread_id, work| + # process_id is a unique identifier per Sidekiq process + # thread_id is a unique identifier per thread + # work is a Hash which looks like: + # {"queue"=>"chaos:chaos_sleep", + # "payload"=> + # { "class"=>"Chaos::SleepWorker", + # "args"=>[1000], + # "retry"=>3, + # "queue"=>"chaos:chaos_sleep", + # "backtrace"=>true, + # "queue_namespace"=>"chaos", + # "jid"=>"b2a31e3eac7b1a99ff235869", + # "created_at"=>1566316974.9215662, + # "correlation_id"=>"e484fb26-7576-45f9-bf21-b99389e1c53c", + # "enqueued_at"=>1566316974.9229589}, + # "run_at"=>1566316974}], +end +``` + +### Remove sidekiq jobs for given parameters (destructive) + +```ruby +# for jobs like this: +# RepositoryImportWorker.new.perform_async(100) +id_list = [100] + +queue = Sidekiq::Queue.new('repository_import') +queue.each do |job| + job.delete if id_list.include?(job.args[0]) +end +``` + +### Remove specific job ID (destructive) + +```ruby +queue = Sidekiq::Queue.new('repository_import') +queue.each do |job| + job.delete if job.jid == 'my-job-id' +end +``` + +## Canceling running jobs (destructive) + +> Introduced in GitLab 12.3. + +This is highly risky operation and use it as last resort. +Doing that might result in data corruption, as the job +is interrupted mid-execution and it is not guaranteed +that proper rollback of transactions is implemented. + +```ruby +Gitlab::SidekiqMonitor.cancel_job('job-id') +``` + +> This requires the Sidekiq to be run with `SIDEKIQ_MONITOR_WORKER=1` +> environment variable. + +To perform of the interrupt we use `Thread.raise` which +has number of drawbacks, as mentioned in [Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): + +> This is where the implications get interesting, and terrifying. This means that an exception can get raised: +> +> * during a network request (ok, as long as the surrounding code is prepared to catch Timeout::Error) +> * during the cleanup for the network request +> * during a rescue block +> * while creating an object to save to the database afterwards +> * in any of your code, regardless of whether it could have possibly raised an exception before +> +> Nobody writes code to defend against an exception being raised on literally any line. That’s not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that’s unlikely :) diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index c6529812ec3..a4bed72b965 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -1,21 +1,18 @@ # Uploads administration ->**Notes:** Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. ## Using local storage ->**Notes:** +NOTE: **Note:** This is the default configuration To change the location where the uploads are stored locally, follow the steps below. ---- - **In Omnibus installations:** ->**Notes:** +NOTE: **Note:** For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It is strongly discouraged to change this configuration option on an existing GitLab installation. _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ @@ -30,8 +27,6 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **In installations from source:** _The uploads are stored by default in @@ -42,8 +37,8 @@ _The uploads are stored by default in ```yaml uploads: - storage_path: /mnt/storage - base_dir: uploads + storage_path: /mnt/storage + base_dir: uploads ``` 1. Save the file and [restart GitLab][] for the changes to take effect. @@ -62,7 +57,7 @@ This configuration relies on valid AWS credentials to be configured already. ## Object Storage Settings -For source installations the following settings are nested under `uploads:` and then `object_store:`. On omnibus installs they are prefixed by `uploads_object_store_`. +For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`. | Setting | Description | Default | |---------|-------------|---------| @@ -83,6 +78,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -108,8 +104,8 @@ _The uploads are stored by default in } ``` - >**Note:** - >If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + NOTE: **Note:** + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. ```ruby gitlab_rails['uploads_object_store_connection'] = { @@ -122,8 +118,6 @@ _The uploads are stored by default in 1. Save the file and [reconfigure GitLab][] for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). ---- - **In installations from source:** _The uploads are stored by default in @@ -147,6 +141,91 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). +### Oracle Cloud S3 connection settings + +Note that Oracle Cloud S3 must be sure to use the following settings: + +| Setting | Value | +|---------|-------| +| `enable_signature_v4_streaming` | false | +| `path_style` | true | + +If `enable_signature_v4_streaming` is set to `true`, you may see the +following error: + +``` +STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported +``` + +### OpenStack compatible connection settings + +The connection settings match those provided by [Fog](https://github.com/fog), and are as follows: + +| Setting | Description | Default | +|---------|-------------|---------| +| `provider` | Always `OpenStack` for compatible hosts | OpenStack | +| `openstack_username` | OpenStack username | | +| `openstack_api_key` | OpenStack api key | | +| `openstack_temp_url_key` | OpenStack key for generating temporary urls | | +| `openstack_auth_url` | OpenStack authentication endpont | | +| `openstack_region` | OpenStack region | | +| `openstack_tenant` | OpenStack tenant ID | + +**In Omnibus installations:** + +_The uploads are stored by default in +`/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._ + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['uploads_object_store_remote_directory'] = "OPENSTACK_OBJECT_CONTAINER_NAME" + gitlab_rails['uploads_object_store_connection'] = { + 'provider' => 'OpenStack', + 'openstack_username' => 'OPENSTACK_USERNAME', + 'openstack_api_key' => 'OPENSTACK_PASSWORD', + 'openstack_temp_url_key' => 'OPENSTACK_TEMP_URL_KEY', + 'openstack_auth_url' => 'https://auth.cloud.ovh.net/v2.0/', + 'openstack_region' => 'DE1', + 'openstack_tenant' => 'TENANT_ID', + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). + +--- + +**In installations from source:** + +_The uploads are stored by default in +`/home/git/gitlab/public/uploads/-/system`._ + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + uploads: + object_store: + enabled: true + direct_upload: false + background_upload: true + proxy_download: false + remote_directory: OPENSTACK_OBJECT_CONTAINER_NAME + connection: + provider: OpenStack + openstack_username: OPENSTACK_USERNAME + openstack_api_key: OPENSTACK_PASSWORD + openstack_temp_url_key: OPENSTACK_TEMP_URL_KEY + openstack_auth_url: 'https://auth.cloud.ovh.net/v2.0/' + openstack_region: DE1 + openstack_tenant: 'TENANT_ID' + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` rake task](raketasks/uploads/migrate.md). + [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" [restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" [eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Premium" diff --git a/doc/api/README.md b/doc/api/README.md index 3ded230370c..33394d46a2d 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -1,151 +1,13 @@ -# GitLab API +# API Docs Automate GitLab via a simple and powerful API. The main GitLab API is a [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) API. Therefore, documentation in this section assumes knowledge of REST concepts. -## API resources - -Available API resources can be grouped in the following contexts: - -- [Projects](#project-resources). -- [Groups](#group-resources). -- [Standalone](#standalone-resources). - -See also: - -- [V3 to V4](v3_to_v4.md). -- Adding [deploy keys for multiple projects](deploy_key_multiple_projects.md). - -### Project resources - -The following API resources are available in the project context: - -| Resource | Available endpoints | -|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | -| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | -| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | -| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | -| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | -| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | -| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | -| [Deployments](deployments.md) | `/projects/:id/deployments` | -| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | -| [Environments](environments.md) | `/projects/:id/environments` | -| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | -| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | -| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | -| [Issue boards](boards.md) | `/projects/:id/boards` | -| [Issue links](issue_links.md) **(STARTER)** | `/projects/:id/issues/.../links` | -| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | -| [Labels](labels.md) | `/projects/:id/labels` | -| [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | -| [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge request approvals](merge_request_approvals.md) **(STARTER)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | -| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | -| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | -| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | -| [Packages](packages.md) **(PREMIUM)** | `/projects/:id/packages` | -| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | -| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | -| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | -| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | -| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | -| [Project badges](project_badges.md) | `/projects/:id/badges` | -| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | -| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | -| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | -| [Project milestones](milestones.md) | `/projects/:id/milestones` | -| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | -| [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | -| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | -| [Releases](releases/index.md) | `/projects/:id/releases` | -| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | -| [Repositories](repositories.md) | `/projects/:id/repository` | -| [Repository files](repository_files.md) | `/projects/:id/repository/files` | -| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | -| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | -| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | -| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | -| [Services](services.md) | `/projects/:id/services` | -| [Tags](tags.md) | `/projects/:id/repository/tags` | -| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` (also available for groups) | -| [Wikis](wikis.md) | `/projects/:id/wikis` | - -### Group resources - -The following API resources are available in the group context: - -| Resource | Available endpoints | -|:-----------------------------------------------------------------|:---------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | -| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | -| [Discussions](discussions.md) (threaded comments) **(ULTIMATE)** | `/groups/:id/epics/.../discussions` (also available for projects) | -| [Epic issues](epic_issues.md) **(ULTIMATE)** | `/groups/:id/epics/.../issues` | -| [Epic links](epic_links.md) **(ULTIMATE)** | `/groups/:id/epics/.../epics` | -| [Epics](epics.md) **(ULTIMATE)** | `/groups/:id/epics` | -| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | -| [Group badges](group_badges.md) | `/groups/:id/badges` | -| [Group issue boards](group_boards.md) | `/groups/:id/boards` | -| [Group labels](group_labels.md) | `/groups/:id/labels` | -| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | -| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | -| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | -| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | -| [Members](members.md) | `/groups/:id/members` (also available for projects) | -| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | -| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | -| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | -| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | -| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | - -### Standalone resources - -The following API resources are available outside of project and group contexts (including `/users`): - -| Resource | Available endpoints | -|:--------------------------------------------------|:------------------------------------------------------------------------| -| [Applications](applications.md) | `/applications` | -| [Avatar](avatar.md) | `/avatar` | -| [Broadcast messages](broadcast_messages.md) | `/broadcast_messages` | -| [Code snippets](snippets.md) | `/snippets` | -| [Custom attributes](custom_attributes.md) | `/users/:id/custom_attributes` (also available for groups and projects) | -| [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | -| [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | -| [Feature flags](features.md) | `/features` | -| [Geo Nodes](geo_nodes.md) **(PREMIUM ONLY)** | `/geo_nodes` | -| [Import repository from GitHub](import.md) | `/import/github` | -| [Issues](issues.md) | `/issues` (also available for groups and projects) | -| [Issues Statistics](issues_statistics.md) | `/issues_statistics` (also available for groups and projects) | -| [Keys](keys.md) | `/keys` | -| [License](license.md) **(CORE ONLY)** | `/license` | -| [Markdown](markdown.md) | `/markdown` | -| [Merge requests](merge_requests.md) | `/merge_requests` (also available for groups and projects) | -| [Namespaces](namespaces.md) | `/namespaces` | -| [Notification settings](notification_settings.md) | `/notification_settings` (also available for groups and projects) | -| [Pages domains](pages_domains.md) | `/pages/domains` (also available for projects) | -| [Projects](projects.md) | `/users/:id/projects` (also available for projects) | -| [Runners](runners.md) | `/runners` (also available for projects) | -| [Search](search.md) | `/search` (also available for groups and projects) | -| [Settings](settings.md) | `/application/settings` | -| [Sidekiq metrics](sidekiq_metrics.md) | `/sidekiq` | -| [Suggestions](suggestions.md) | `/suggestions` | -| [System hooks](system_hooks.md) | `/hooks` | -| [Todos](todos.md) | `/todos` | -| [Users](users.md) | `/users` | -| [Validate `.gitlab-ci.yml` file](lint.md) | `/lint` | -| [Version](version.md) | `/version` | - -### Templates API resources - -Endpoints are available for: - -- [Dockerfile templates](templates/dockerfiles.md). -- [`.gitignore` templates](templates/gitignores.md). -- [GitLab CI YAML templates](templates/gitlab_ci_ymls.md). -- [Open source license templates](templates/licenses.md). +## Available API resources + +For a list of the available resources and their endpoints, see +[API resources](api_resources.md). ## SCIM **(SILVER ONLY)** @@ -154,9 +16,10 @@ the `/Users` endpoint. The base URL is: `/api/scim/v2/groups/:group_path/Users/` ## Road to GraphQL -Going forward, we will start on moving to -[GraphQL](graphql/index.md) and deprecate the use of -controller-specific endpoints. GraphQL has a number of benefits: +[GraphQL](graphql/index.md) is available in GitLab, which will +allow deprecation of controller-specific endpoints. + +GraphQL has a number of benefits: 1. We avoid having to maintain two different APIs. 1. Callers of the API can request only what they need. @@ -214,11 +77,12 @@ authentication is not provided. For those cases where it is not required, this will be mentioned in the documentation for each individual endpoint. For example, the [`/projects/:id` endpoint](projects.md). -There are three ways to authenticate with the GitLab API: +There are four ways to authenticate with the GitLab API: 1. [OAuth2 tokens](#oauth2-tokens) 1. [Personal access tokens](#personal-access-tokens) 1. [Session cookie](#session-cookie) +1. [GitLab CI job token](#gitlab-ci-job-token-premium) **(PREMIUM)** For admins who want to authenticate with the API as a specific user, or who want to build applications or scripts that do so, two options are available: @@ -270,6 +134,12 @@ Example of using the personal access token in a header: curl --header "Private-Token: <your_access_token>" https://gitlab.example.com/api/v4/projects ``` +You can also use personal access tokens with OAuth-compliant headers: + +```shell +curl --header "Authorization: Bearer <your_access_token>" https://gitlab.example.com/api/v4/projects +``` + Read more about [personal access tokens][pat]. ### Session cookie @@ -282,6 +152,14 @@ The primary user of this authentication method is the web frontend of GitLab its which can use the API as the authenticated user to get a list of their projects, for example, without needing to explicitly pass an access token. +### GitLab CI job token **(PREMIUM)** + +With a few API endpoints you can use a [GitLab CI job token](../user/project/new_ci_build_permissions_model.md#job-token) +to authenticate with the API: + +- [Get job artifacts](jobs.md#get-job-artifacts) +- [Pipeline triggers](pipeline_triggers.md) + ### Impersonation tokens > [Introduced][ce-9099] in GitLab 9.0. Needs admin permissions. @@ -320,8 +198,6 @@ By default, impersonation is enabled. To disable impersonation: To re-enable impersonation, remove this configuration and reconfigure GitLab. ---- - **For installations from source** 1. Edit `config/gitlab.yml`: @@ -380,8 +256,6 @@ returned with status code `404`: } ``` ---- - Example of a valid API call and a request using cURL with sudo request, providing a username: @@ -511,7 +385,7 @@ more than 10,000, the `X-Total` and `X-Total-Pages` headers as well as the ## Namespaced path encoding -If using namespaced API calls, make sure that the `NAMESPACE/PROJECT_NAME` is +If using namespaced API calls, make sure that the `NAMESPACE/PROJECT_PATH` is URL-encoded. For example, `/` is represented by `%2F`: @@ -520,6 +394,11 @@ For example, `/` is represented by `%2F`: GET /api/v4/projects/diaspora%2Fdiaspora ``` +NOTE: **Note:** +A project's **path** is not necessarily the same as its **name**. A +project's path can found in the project's URL or in the project's settings +under **General > Advanced > Change path**. + ## Branches and tags name encoding If your branch or tag contains a `/`, make sure the branch/tag name is @@ -686,6 +565,13 @@ The correct encoding for the query parameter would be: There are many unofficial GitLab API Clients for most of the popular programming languages. Visit the [GitLab website] for a complete list. +## Rate limits + +For administrator documentation on rate limit settings, see +[Rate limits](../security/rate_limits.md). To find the settings that are +specifically used by GitLab.com, see +[GitLab.com-specific rate limits](../user/gitlab_com/index.md#gitlabcom-specific-rate-limits). + [GitLab website]: https://about.gitlab.com/applications/#api-clients "Clients using the GitLab API" [lib-api-url]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/api/api.rb [ce-3749]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3749 diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md new file mode 100644 index 00000000000..9af5430f1c8 --- /dev/null +++ b/doc/api/api_resources.md @@ -0,0 +1,144 @@ +# API resources + +Available resources for the [GitLab API](README.md) can be grouped in the following contexts: + +- [Projects](#project-resources). +- [Groups](#group-resources). +- [Standalone](#standalone-resources). + +See also: + +- [V3 to V4](v3_to_v4.md). +- Adding [deploy keys for multiple projects](deploy_key_multiple_projects.md). +- [API Resources for various templates](#templates-api-resources). + +## Project resources + +The following API resources are available in the project context: + +| Resource | Available endpoints | +|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | +| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | +| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | +| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | +| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | +| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | +| [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` +| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | +| [Deployments](deployments.md) | `/projects/:id/deployments` | +| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | +| [Environments](environments.md) | `/projects/:id/environments` | +| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | +| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | +| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | +| [Issue boards](boards.md) | `/projects/:id/boards` | +| [Issue links](issue_links.md) **(STARTER)** | `/projects/:id/issues/.../links` | +| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | +| [Labels](labels.md) | `/projects/:id/labels` | +| [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | +| [Members](members.md) | `/projects/:id/members` (also available for groups) | +| [Merge request approvals](merge_request_approvals.md) **(STARTER)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | +| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | +| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | +| [Packages](packages.md) **(PREMIUM)** | `/projects/:id/packages` | +| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | +| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | +| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | +| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | +| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | +| [Project badges](project_badges.md) | `/projects/:id/badges` | +| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | +| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | +| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | +| [Project milestones](milestones.md) | `/projects/:id/milestones` | +| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | +| [Project templates](project_templates.md) | `/projects/:id/templates` | +| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | +| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | +| [Releases](releases/index.md) | `/projects/:id/releases` | +| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | +| [Repositories](repositories.md) | `/projects/:id/repository` | +| [Repository files](repository_files.md) | `/projects/:id/repository/files` | +| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | +| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | +| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | +| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | +| [Services](services.md) | `/projects/:id/services` | +| [Tags](tags.md) | `/projects/:id/repository/tags` | +| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` +| [Wikis](wikis.md) | `/projects/:id/wikis` | + +## Group resources + +The following API resources are available in the group context: + +| Resource | Available endpoints | +|:-----------------------------------------------------------------|:---------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | +| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | +| [Discussions](discussions.md) (threaded comments) **(ULTIMATE)** | `/groups/:id/epics/.../discussions` (also available for projects) | +| [Epic issues](epic_issues.md) **(ULTIMATE)** | `/groups/:id/epics/.../issues` | +| [Epic links](epic_links.md) **(ULTIMATE)** | `/groups/:id/epics/.../epics` | +| [Epics](epics.md) **(ULTIMATE)** | `/groups/:id/epics` | +| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | +| [Group badges](group_badges.md) | `/groups/:id/badges` | +| [Group issue boards](group_boards.md) | `/groups/:id/boards` | +| [Group labels](group_labels.md) | `/groups/:id/labels` | +| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | +| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | +| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | +| [Issues Statistics](issues_statistics.md) | `/groups/:id/issues_statistics` (also available for projects and standalone) | +| [Members](members.md) | `/groups/:id/members` (also available for projects) | +| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | +| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | +| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | +| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | +| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | + +## Standalone resources + +The following API resources are available outside of project and group contexts (including `/users`): + +| Resource | Available endpoints | +|:--------------------------------------------------|:------------------------------------------------------------------------| +| [Applications](applications.md) | `/applications` | +| [Avatar](avatar.md) | `/avatar` | +| [Broadcast messages](broadcast_messages.md) | `/broadcast_messages` | +| [Code snippets](snippets.md) | `/snippets` | +| [Custom attributes](custom_attributes.md) | `/users/:id/custom_attributes` (also available for groups and projects) | +| [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | +| [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | +| [Feature flags](features.md) | `/features` | +| [Geo Nodes](geo_nodes.md) **(PREMIUM ONLY)** | `/geo_nodes` | +| [Import repository from GitHub](import.md) | `/import/github` | +| [Issues](issues.md) | `/issues` (also available for groups and projects) | +| [Issues Statistics](issues_statistics.md) | `/issues_statistics` (also available for groups and projects) | +| [Keys](keys.md) | `/keys` | +| [License](license.md) **(CORE ONLY)** | `/license` | +| [Markdown](markdown.md) | `/markdown` | +| [Merge requests](merge_requests.md) | `/merge_requests` (also available for groups and projects) | +| [Namespaces](namespaces.md) | `/namespaces` | +| [Notification settings](notification_settings.md) | `/notification_settings` (also available for groups and projects) | +| [Pages domains](pages_domains.md) | `/pages/domains` (also available for projects) | +| [Projects](projects.md) | `/users/:id/projects` (also available for projects) | +| [Runners](runners.md) | `/runners` (also available for projects) | +| [Search](search.md) | `/search` (also available for groups and projects) | +| [Settings](settings.md) | `/application/settings` | +| [Sidekiq metrics](sidekiq_metrics.md) | `/sidekiq` | +| [Suggestions](suggestions.md) | `/suggestions` | +| [System hooks](system_hooks.md) | `/hooks` | +| [Todos](todos.md) | `/todos` | +| [Users](users.md) | `/users` | +| [Validate `.gitlab-ci.yml` file](lint.md) | `/lint` | +| [Version](version.md) | `/version` | + +## Templates API resources + +Endpoints are available for: + +- [Dockerfile templates](templates/dockerfiles.md). +- [`.gitignore` templates](templates/gitignores.md). +- [GitLab CI YAML templates](templates/gitlab_ci_ymls.md). +- [Open source license templates](templates/licenses.md). diff --git a/doc/api/boards.md b/doc/api/boards.md index 08ec1d832df..b848d7788cd 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -466,7 +466,7 @@ Example response: ## Delete a board list -Only for admins and project owners. Soft deletes the board list in question. +Only for admins and project owners. Deletes the board list in question. ``` DELETE /projects/:id/boards/:board_id/lists/:list_id diff --git a/doc/api/commits.md b/doc/api/commits.md index 25015fad9e3..1f17eaea46d 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -72,15 +72,16 @@ POST /projects/:id/repository/commits | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | -| `branch` | string | yes | Name of the branch to commit into. To create a new branch, also provide `start_branch`. | +| `branch` | string | yes | Name of the branch to commit into. To create a new branch, also provide either `start_branch` or `start_sha`, and optionally `start_project`. | | `commit_message` | string | yes | Commit message | -| `start_branch` | string | no | Name of the branch to start the new commit from | -| `start_project` | integer/string | no | The project ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) to start the commit from. Defaults to the value of `id`. | +| `start_branch` | string | no | Name of the branch to start the new branch from | +| `start_sha` | string | no | SHA of the commit to start the new branch from | +| `start_project` | integer/string | no | The project ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) to start the new branch from. Defaults to the value of `id`. | | `actions[]` | array | yes | An array of action hashes to commit as a batch. See the next table for what attributes it can take. | | `author_email` | string | no | Specify the commit author's email address | | `author_name` | string | no | Specify the commit author's name | | `stats` | boolean | no | Include commit stats. Default is true | -| `force` | boolean | no | When `true` overwrites the target branch with a new commit based on the `start_branch` | +| `force` | boolean | no | When `true` overwrites the target branch with a new commit based on the `start_branch` or `start_sha` | | `actions[]` Attribute | Type | Required | Description | | --------------------- | ---- | -------- | ----------- | @@ -581,6 +582,7 @@ POST /projects/:id/statuses/:sha | `target_url` | string | no | The target URL to associate with this status | `description` | string | no | The short description of the status | `coverage` | float | no | The total code coverage +| `pipeline_id` | integer | no | The ID of the pipeline to set status. Use in case of several pipeline on same SHA. ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/17/statuses/18f3e63d05582537db6d183d9d557be09e1f90c8?state=success" @@ -716,6 +718,7 @@ Example response if commit is signed: ``` Example response if commit is unsigned: + ```json { "message": "404 GPG Signature Not Found" diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index 64ea15bca93..bf544f64178 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -6,6 +6,8 @@ This is the API docs of the [GitLab Container Registry](../user/project/containe ## List registry repositories +### Within a project + Get a list of registry repositories in a project. ``` @@ -14,7 +16,8 @@ GET /projects/:id/registry/repositories | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) accessible by the authenticated user. | +| `tags` | boolean | no | If the param is included as true, each repository will include an array of `"tags"` in the response. | ```bash curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories" @@ -28,6 +31,7 @@ Example response: "id": 1, "name": "", "path": "group/project", + "project_id": 9, "location": "gitlab.example.com:5000/group/project", "created_at": "2019-01-10T13:38:57.391Z" }, @@ -35,12 +39,77 @@ Example response: "id": 2, "name": "releases", "path": "group/project/releases", + "project_id": 9, "location": "gitlab.example.com:5000/group/project/releases", "created_at": "2019-01-10T13:39:08.229Z" } ] ``` +### Within a group + +Get a list of registry repositories in a group. + +``` +GET /groups/:id/registry/repositories +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) accessible by the authenticated user. | +| `tags` | boolean | no | If the param is included as true, each repository will include an array of `"tags"` in the response. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/2/registry/repositories?tags=1" +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "", + "path": "group/project", + "project_id": 9, + "location": "gitlab.example.com:5000/group/project", + "created_at": "2019-01-10T13:38:57.391Z", + "tags": [ + { + "name": "0.0.1", + "path": "group/project:0.0.1", + "location": "gitlab.example.com:5000/group/project:0.0.1" + } + ] + }, + { + "id": 2, + "name": "", + "path": "group/other_project", + "project_id": 11, + "location": "gitlab.example.com:5000/group/other_project", + "created_at": "2019-01-10T13:39:08.229Z", + "tags": [ + { + "name": "0.0.1", + "path": "group/other_project:0.0.1", + "location": "gitlab.example.com:5000/group/other_project:0.0.1" + }, + { + "name": "0.0.2", + "path": "group/other_project:0.0.2", + "location": "gitlab.example.com:5000/group/other_project:0.0.2" + }, + { + "name": "latest", + "path": "group/other_project:latest", + "location": "gitlab.example.com:5000/group/other_project:latest" + } + ] + } +] +``` + ## Delete registry repository Delete a repository in registry. @@ -62,6 +131,8 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git ## List repository tags +### Within a project + Get a list of tags for given registry repository. ``` @@ -70,7 +141,7 @@ GET /projects/:id/registry/repositories/:repository_id/tags | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) accessible by the authenticated user. | | `repository_id` | integer | yes | The ID of registry repository. | ```bash @@ -104,7 +175,7 @@ GET /projects/:id/registry/repositories/:repository_id/tags/:tag_name | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) accessible by the authenticated user. | | `repository_id` | integer | yes | The ID of registry repository. | | `tag_name` | string | yes | The name of tag. | @@ -193,13 +264,13 @@ Examples: curl --request DELETE --data 'name_regex=[0-9a-z]{40}' --data 'keep_n=5' --data 'older_than=2d' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` -2. Remove all tags, but keep always the latest 5: +1. Remove all tags, but keep always the latest 5: ```bash curl --request DELETE --data 'name_regex=.*' --data 'keep_n=5' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` -3. Remove all tags that are older than 1 month: +1. Remove all tags that are older than 1 month: ```bash curl --request DELETE --data 'name_regex=.*' --data 'older_than=1month' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" diff --git a/doc/api/dependencies.md b/doc/api/dependencies.md new file mode 100644 index 00000000000..5296d4e316f --- /dev/null +++ b/doc/api/dependencies.md @@ -0,0 +1,56 @@ +# Dependencies API **(ULTIMATE)** + +CAUTION: **Caution:** +This API is in an alpha stage and considered unstable. +The response payload may be subject to change or breakage +across GitLab releases. + +Every call to this endpoint requires authentication. To perform this call, user should be authorized to read repository. +To see vulnerabilities in response, user should be authorized to read +[Project Security Dashboard](../user/application_security/security_dashboard/index.md#project-security-dashboard). + +## List project dependencies + +Get a list of project dependencies. This API partially mirroring +[Dependency List](../user/application_security/dependency_list/index.md) feature. +This list can be generated only for [languages and package managers](../user/application_security/dependency_scanning/index.md#supported-languages-and-package-managers) +supported by Gemnasium. + +``` +GET /projects/:id/dependencies +GET /projects/:id/dependencies?package_manager=maven +GET /projects/:id/dependencies?package_manager=yarn,bundler +``` + +| Attribute | Type | Required | Description | +| ------------- | -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `package_manager` | string array | no | Returns dependencies belonging to specified package manager. Valid values: `bundler`, `composer`, `maven`, `npm`, `pip` or `yarn`. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/dependencies +``` + +Example response: + +```json +[ + { + "name": "rails", + "version": "5.0.1", + "package_manager": "bundler", + "dependency_file_path": "Gemfile.lock", + "vulnerabilities": [{ + "name": "DDoS", + "severity": "unknown" + }] + }, + { + "name": "hanami", + "version": "1.3.1", + "package_manager": "bundler", + "dependency_file_path": "Gemfile.lock", + "vulnerabilities": [] + } +] +``` diff --git a/doc/api/deploy_key_multiple_projects.md b/doc/api/deploy_key_multiple_projects.md index 0c9e3e66cae..85df972746e 100644 --- a/doc/api/deploy_key_multiple_projects.md +++ b/doc/api/deploy_key_multiple_projects.md @@ -1,29 +1,5 @@ -# Adding deploy keys to multiple projects via API +--- +redirect_to: deploy_keys.md#adding-deploy-keys-to-multiple-projects +--- -If you want to easily add the same deploy key to multiple projects in the same -group, this can be achieved quite easily with the API. - -First, find the ID of the projects you're interested in, by either listing all -projects: - -``` -curl --header 'PRIVATE-TOKEN: <your_access_token>' https://gitlab.example.com/api/v4/projects -``` - -Or finding the ID of a group and then listing all projects in that group: - -``` -curl --header 'PRIVATE-TOKEN: <your_access_token>' https://gitlab.example.com/api/v4/groups - -# For group 1234: -curl --header 'PRIVATE-TOKEN: <your_access_token>' https://gitlab.example.com/api/v4/groups/1234 -``` - -With those IDs, add the same deploy key to all: - -``` -for project_id in 321 456 987; do - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"title": "my key", "key": "ssh-rsa AAAA..."}' https://gitlab.example.com/api/v4/projects/${project_id}/deploy_keys -done -``` +This document was moved to [another location](deploy_keys.md#adding-deploy-keys-to-multiple-projects). diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index 41f6ab436e8..df8cf06fc4a 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -203,3 +203,33 @@ Example response: "created_at" : "2015-08-29T12:44:31.550Z" } ``` + +## Adding deploy keys to multiple projects + +If you want to easily add the same deploy key to multiple projects in the same +group, this can be achieved quite easily with the API. + +First, find the ID of the projects you're interested in, by either listing all +projects: + +``` +curl --header 'PRIVATE-TOKEN: <your_access_token>' https://gitlab.example.com/api/v4/projects +``` + +Or finding the ID of a group and then listing all projects in that group: + +``` +curl --header 'PRIVATE-TOKEN: <your_access_token>' https://gitlab.example.com/api/v4/groups + +# For group 1234: +curl --header 'PRIVATE-TOKEN: <your_access_token>' https://gitlab.example.com/api/v4/groups/1234 +``` + +With those IDs, add the same deploy key to all: + +``` +for project_id in 321 456 987; do + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"title": "my key", "key": "ssh-rsa AAAA..."}' https://gitlab.example.com/api/v4/projects/${project_id}/deploy_keys +done +``` diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 208b8dca2e2..12dbba78291 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -12,16 +12,15 @@ This includes system notes, which are notes about changes to the object (for exa ## Discussions pagination -By default, `GET` requests return 20 results at a time because the API results -are paginated. +By default, `GET` requests return 20 results at a time because the API results are paginated. Read more on [pagination](README.md#pagination). ## Issues -### List project issue discussions +### List project issue discussion items -Gets a list of all discussions for a single issue. +Gets a list of all discussion items for a single issue. ``` GET /projects/:id/issues/:issue_iid/discussions @@ -115,9 +114,9 @@ GET /projects/:id/issues/:issue_iid/discussions curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions ``` -### Get single issue discussion +### Get single issue discussion item -Returns a single discussion for a specific project issue +Returns a single discussion item for a specific project issue ``` GET /projects/:id/issues/:issue_iid/discussions/:discussion_id @@ -129,16 +128,15 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | -| `discussion_id` | integer | yes | The ID of a discussion | +| `discussion_id` | integer | yes | The ID of a discussion item | ```bash curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 ``` -### Create new issue discussion +### Create new issue thread -Creates a new discussion to a single project issue. This is similar to creating -a note but other comments (replies) can be added to it later. +Creates a new thread to a single project issue. This is similar to creating a note but other comments (replies) can be added to it later. ``` POST /projects/:id/issues/:issue_iid/discussions @@ -150,17 +148,19 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | -| `body` | string | yes | The content of a discussion | +| `body` | string | yes | The content of the thread | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions?body=comment ``` -### Add note to existing issue discussion +### Add note to existing issue thread + +Adds a new note to the thread. This can also [create a thread from a single comment](../user/discussions/#start-a-thread-by-replying-to-a-standard-comment). -Adds a new note to the discussion. This can also -[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). +**WARNING** +Notes can be added to other items than comments (system notes, etc.) making them threads. ``` POST /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes @@ -172,18 +172,18 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment ``` -### Modify existing issue discussion note +### Modify existing issue thread note -Modify existing discussion note of an issue. +Modify existing thread note of an issue. ``` PUT /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes/:note_id @@ -195,17 +195,17 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | ```bash curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment ``` -### Delete an issue discussion note +### Delete an issue thread note -Deletes an existing discussion note of an issue. +Deletes an existing thread note of an issue. ``` DELETE /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes/:note_id @@ -226,9 +226,9 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitl ## Snippets -### List project snippet discussions +### List project snippet discussion items -Gets a list of all discussions for a single snippet. +Gets a list of all discussion items for a single snippet. ``` GET /projects/:id/snippets/:snippet_id/discussions @@ -322,9 +322,9 @@ GET /projects/:id/snippets/:snippet_id/discussions curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions ``` -### Get single snippet discussion +### Get single snippet discussion item -Returns a single discussion for a specific project snippet +Returns a single discussion item for a specific project snippet ``` GET /projects/:id/snippets/:snippet_id/discussions/:discussion_id @@ -336,15 +336,15 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `snippet_id` | integer | yes | The ID of an snippet | -| `discussion_id` | integer | yes | The ID of a discussion | +| `discussion_id` | integer | yes | The ID of a discussion item | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 ``` -### Create new snippet discussion +### Create new snippet thread -Creates a new discussion to a single project snippet. This is similar to creating +Creates a new thread to a single project snippet. This is similar to creating a note but other comments (replies) can be added to it later. ``` @@ -364,9 +364,9 @@ Parameters: curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions?body=comment ``` -### Add note to existing snippet discussion +### Add note to existing snippet thread -Adds a new note to the discussion. +Adds a new note to the thread. ``` POST /projects/:id/snippets/:snippet_id/discussions/:discussion_id/notes @@ -378,18 +378,18 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `snippet_id` | integer | yes | The ID of an snippet | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment ``` -### Modify existing snippet discussion note +### Modify existing snippet thread note -Modify existing discussion note of an snippet. +Modify existing thread note of a snippet. ``` PUT /projects/:id/snippets/:snippet_id/discussions/:discussion_id/notes/:note_id @@ -401,17 +401,17 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `snippet_id` | integer | yes | The ID of an snippet | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | ```bash curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment ``` -### Delete an snippet discussion note +### Delete a snippet thread note -Deletes an existing discussion note of an snippet. +Deletes an existing thread note of a snippet. ``` DELETE /projects/:id/snippets/:snippet_id/discussions/:discussion_id/notes/:note_id @@ -432,9 +432,9 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitl ## Epics **(ULTIMATE)** -### List group epic discussions +### List group epic discussion items -Gets a list of all discussions for a single epic. +Gets a list of all discussion items for a single epic. ``` GET /groups/:id/epics/:epic_id/discussions @@ -529,9 +529,9 @@ GET /groups/:id/epics/:epic_id/discussions curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions ``` -### Get single epic discussion +### Get single epic discussion item -Returns a single discussion for a specific group epic +Returns a single discussion item for a specific group epic ``` GET /groups/:id/epics/:epic_id/discussions/:discussion_id @@ -543,16 +543,16 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | -| `discussion_id` | integer | yes | The ID of a discussion | +| `discussion_id` | integer | yes | The ID of a discussion item | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 ``` -### Create new epic discussion +### Create new epic thread -Creates a new discussion to a single group epic. This is similar to creating -a note but but another comments (replies) can be added to it later. +Creates a new thread to a single group epic. This is similar to creating +a note but but other comments (replies) can be added to it later. ``` POST /groups/:id/epics/:epic_id/discussions @@ -564,17 +564,17 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | -| `body` | string | yes | The content of a discussion | +| `body` | string | yes | The content of the thread | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions?body=comment ``` -### Add note to existing epic discussion +### Add note to existing epic thread -Adds a new note to the discussion. This can also -[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). +Adds a new note to the thread. This can also +[create a thread from a single comment](../user/discussions/#start-a-thread-by-replying-to-a-standard-comment). ``` POST /groups/:id/epics/:epic_id/discussions/:discussion_id/notes @@ -585,19 +585,19 @@ Parameters: | Attribute | Type | Required | Description | | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | -| `epic_id` | integer | yes | The ID of an epic | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment ``` -### Modify existing epic discussion note +### Modify existing epic thread note -Modify existing discussion note of an epic. +Modify existing thread note of an epic. ``` PUT /groups/:id/epics/:epic_id/discussions/:discussion_id/notes/:note_id @@ -609,17 +609,17 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of note/reply | ```bash curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment ``` -### Delete an epic discussion note +### Delete an epic thread note -Deletes an existing discussion note of an epic. +Deletes an existing thread note of an epic. ``` DELETE /groups/:id/epics/:epic_id/discussions/:discussion_id/notes/:note_id @@ -631,8 +631,8 @@ Parameters: | --------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/636 @@ -640,9 +640,9 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitl ## Merge requests -### List project merge request discussions +### List project merge request discussion items -Gets a list of all discussions for a single merge request. +Gets a list of all discussion items for a single merge request. ``` GET /projects/:id/merge_requests/:merge_request_iid/discussions @@ -789,9 +789,9 @@ Diff comments contain also position: curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions ``` -### Get single merge request discussion +### Get single merge request discussion item -Returns a single discussion for a specific project merge request +Returns a single discussion item for a specific project merge request ``` GET /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id @@ -803,15 +803,15 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | -| `discussion_id` | integer | yes | The ID of a discussion | +| `discussion_id` | integer | yes | The ID of a discussion item | ```bash curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 ``` -### Create new merge request discussion +### Create new merge request thread -Creates a new discussion to a single project merge request. This is similar to creating +Creates a new thread to a single project merge request. This is similar to creating a note but other comments (replies) can be added to it later. ``` @@ -824,7 +824,7 @@ Parameters: | ------------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | -| `body` | string | yes | The content of a discussion | +| `body` | string | yes | The content of the thread | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | | `position` | hash | no | Position when creating a diff note | | `position[base_sha]` | string | yes | Base commit SHA in the source branch | @@ -844,9 +844,9 @@ Parameters: curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions?body=comment ``` -### Resolve a merge request discussion +### Resolve a merge request thread -Resolve/unresolve whole discussion of a merge request. +Resolve/unresolve whole thread of a merge request. ``` PUT /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id @@ -858,17 +858,17 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | -| `discussion_id` | integer | yes | The ID of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | | `resolved` | boolean | yes | Resolve/unresolve the discussion | ```bash curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7?resolved=true ``` -### Add note to existing merge request discussion +### Add note to existing merge request thread -Adds a new note to the discussion. This can also -[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). +Adds a new note to the thread. This can also +[create a thread from a single comment](../user/discussions/#start-a-thread-by-replying-to-a-standard-comment). ``` POST /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id/notes @@ -880,18 +880,18 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment ``` -### Modify an existing merge request discussion note +### Modify an existing merge request thread note -Modify or resolve an existing discussion note of a merge request. +Modify or resolve an existing thread note of a merge request. ``` PUT /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id/notes/:note_id @@ -903,9 +903,9 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | no | The content of a discussion (exactly one of `body` or `resolved` must be set | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | no | The content of the note/reply (exactly one of `body` or `resolved` must be set | | `resolved` | boolean | no | Resolve/unresolve the note (exactly one of `body` or `resolved` must be set | ```bash @@ -918,9 +918,9 @@ Resolving a note: curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?resolved=true ``` -### Delete a merge request discussion note +### Delete a merge request thread note -Deletes an existing discussion note of a merge request. +Deletes an existing thread note of a merge request. ``` DELETE /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id/notes/:note_id @@ -932,8 +932,8 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of a merge request | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions/636 @@ -941,9 +941,9 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitl ## Commits -### List project commit discussions +### List project commit discussion items -Gets a list of all discussions for a single commit. +Gets a list of all discussion items for a single commit. ``` GET /projects/:id/commits/:commit_id/discussions @@ -1082,9 +1082,9 @@ Diff comments contain also position: curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/commits/11/discussions ``` -### Get single commit discussion +### Get single commit discussion item -Returns a single discussion for a specific project commit +Returns a single discussion item for a specific project commit ``` GET /projects/:id/commits/:commit_id/discussions/:discussion_id @@ -1096,15 +1096,15 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `commit_id` | integer | yes | The ID of a commit | -| `discussion_id` | integer | yes | The ID of a discussion | +| `discussion_id` | integer | yes | The ID of a discussion item | ```bash curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/commits/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 ``` -### Create new commit discussion +### Create new commit thread -Creates a new discussion to a single project commit. This is similar to creating +Creates a new thread to a single project commit. This is similar to creating a note but other comments (replies) can be added to it later. ``` @@ -1117,7 +1117,7 @@ Parameters: | ------------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `commit_id` | integer | yes | The ID of a commit | -| `body` | string | yes | The content of a discussion | +| `body` | string | yes | The content of the thread | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | | `position` | hash | no | Position when creating a diff note | | `position[base_sha]` | string | yes | Base commit SHA in the source branch | @@ -1137,9 +1137,9 @@ Parameters: curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/commits/11/discussions?body=comment ``` -### Add note to existing commit discussion +### Add note to existing commit thread -Adds a new note to the discussion. +Adds a new note to the thread. ``` POST /projects/:id/commits/:commit_id/discussions/:discussion_id/notes @@ -1151,18 +1151,18 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `commit_id` | integer | yes | The ID of a commit | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | -| `body` | string | yes | The content of a discussion | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | +| `body` | string | yes | The content of the note/reply | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/commits/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment ``` -### Modify an existing commit discussion note +### Modify an existing commit thread note -Modify or resolve an existing discussion note of a commit. +Modify or resolve an existing thread note of a commit. ``` PUT /projects/:id/commits/:commit_id/discussions/:discussion_id/notes/:note_id @@ -1174,8 +1174,8 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `commit_id` | integer | yes | The ID of a commit | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | | `body` | string | no | The content of a note | ```bash @@ -1188,9 +1188,9 @@ Resolving a note: curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/commits/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?resolved=true ``` -### Delete a commit discussion note +### Delete a commit thread note -Deletes an existing discussion note of a commit. +Deletes an existing thread note of a commit. ``` DELETE /projects/:id/commits/:commit_id/discussions/:discussion_id/notes/:note_id @@ -1202,8 +1202,8 @@ Parameters: | ------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `commit_id` | integer | yes | The ID of a commit | -| `discussion_id` | integer | yes | The ID of a discussion | -| `note_id` | integer | yes | The ID of a discussion note | +| `discussion_id` | integer | yes | The ID of a thread | +| `note_id` | integer | yes | The ID of a thread note | ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/commits/11/discussions/636 diff --git a/doc/api/epics.md b/doc/api/epics.md index d05eb0a8804..87ae0c48199 100644 --- a/doc/api/epics.md +++ b/doc/api/epics.md @@ -10,7 +10,7 @@ If epics feature is not available a `403` status code will be returned. The [epic issues API](epic_issues.md) allows you to interact with issues associated with an epic. -# Milestone dates integration +## Milestone dates integration > [Introduced][ee-6448] in GitLab 11.3. @@ -65,6 +65,8 @@ Example response: "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/4", + "reference": "&4", "author": { "id": 10, "name": "Lu Mayer", @@ -118,6 +120,8 @@ Example response: "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/5", + "reference": "&5", "author":{ "id": 7, "name": "Pamella Huel", @@ -182,6 +186,8 @@ Example response: "title": "Epic", "description": "Epic description", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/6", + "reference": "&6", "author": { "name" : "Alexandra Bashirian", "avatar_url" : null, @@ -247,6 +253,8 @@ Example response: "title": "New Title", "description": "Epic description", "state": "opened", + "web_edit_url": "http://localhost:3001/groups/test/-/epics/6", + "reference": "&6", "author": { "name" : "Alexandra Bashirian", "avatar_url" : null, diff --git a/doc/api/events.md b/doc/api/events.md index 6dca8e52f69..1cd7047b867 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -70,7 +70,7 @@ Parameters: Example request: -``` +```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/events?target_type=issue&action=created&after=2017-01-31&before=2017-03-01 ``` @@ -275,7 +275,7 @@ Parameters: Example request: -``` +```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:project_id/events?target_type=issue&action=created&after=2017-01-31&before=2017-03-01 ``` @@ -343,8 +343,8 @@ Example response: "username": "root", "id": 1, "state": "active", - "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/root" + "avatar_url": "https://gitlab.example.com/uploads/user/avatar/1/fox_avatar.png", + "web_url": "https://gitlab.example.com/root" }, "created_at": "2015-12-04T10:33:56.698Z", "system": false, @@ -357,8 +357,8 @@ Example response: "username": "root", "id": 1, "state": "active", - "avatar_url": "http://localhost:3000/uploads/user/avatar/1/fox_avatar.png", - "web_url": "http://localhost:3000/root" + "avatar_url": "https://gitlab.example.com/uploads/user/avatar/1/fox_avatar.png", + "web_url": "https://gitlab.example.com/root" }, "author_username": "root" } diff --git a/doc/api/features.md b/doc/api/features.md index 6ecd4ec14b9..e8d0c7c942b 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -60,8 +60,8 @@ POST /features/:name | `value` | integer/string | yes | `true` or `false` to enable/disable, or an integer for percentage of time | | `feature_group` | string | no | A Feature group name | | `user` | string | no | A GitLab username | -| `group` | string | no | A GitLab group's path, for example 'gitlab-org' | -| `project` | string | no | A projects path, for example 'gitlab-org/gitlab-ce' | +| `group` | string | no | A GitLab group's path, for example `gitlab-org` | +| `project` | string | no | A projects path, for example `gitlab-org/gitlab-ce` | Note that you can enable or disable a feature for a `feature_group`, a `user`, a `group`, and a `project` in a single API call. diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index ac64cbedf7d..5eba7f038ed 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -10,7 +10,7 @@ GET /geo_nodes ``` ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes +curl --header "PRIVATE-TOKEN: <your_access_token>" https://primary.example.com/api/v4/geo_nodes ``` Example response: @@ -27,8 +27,15 @@ Example response: "current": true, "files_max_capacity": 10, "repos_max_capacity": 25, + "container_repositories_max_capacity": 10, "verification_max_capacity": 100, - "clone_protocol": "http" + "clone_protocol": "http", + "web_edit_url": "https://primary.example.com/admin/geo/nodes/1/edit", + "_links": { + "self": "https://primary.example.com/api/v4/geo_nodes/1", + "status":"https://primary.example.com/api/v4/geo_nodes/1/status", + "repair":"https://primary.example.com/api/v4/geo_nodes/1/repair" + } }, { "id": 2, @@ -40,8 +47,17 @@ Example response: "current": false, "files_max_capacity": 10, "repos_max_capacity": 25, + "container_repositories_max_capacity": 10, "verification_max_capacity": 100, - "clone_protocol": "http" + "sync_object_storage": true, + "clone_protocol": "http", + "web_edit_url": "https://primary.example.com/admin/geo/nodes/2/edit", + "web_geo_projects_url": "https://secondary.example.com/admin/geo/projects", + "_links": { + "self":"https://primary.example.com/api/v4/geo_nodes/2", + "status":"https://primary.example.com/api/v4/geo_nodes/2/status", + "repair":"https://primary.example.com/api/v4/geo_nodes/2/repair" + } } ] ``` @@ -53,7 +69,7 @@ GET /geo_nodes/:id ``` ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/1 +curl --header "PRIVATE-TOKEN: <your_access_token>" https://primary.example.com/api/v4/geo_nodes/1 ``` Example response: @@ -69,8 +85,15 @@ Example response: "current": true, "files_max_capacity": 10, "repos_max_capacity": 25, + "container_repositories_max_capacity": 10, "verification_max_capacity": 100, - "clone_protocol": "http" + "clone_protocol": "http", + "web_edit_url": "https://primary.example.com/admin/geo/nodes/1/edit", + "_links": { + "self": "https://primary.example.com/api/v4/geo_nodes/1", + "status":"https://primary.example.com/api/v4/geo_nodes/1/status", + "repair":"https://primary.example.com/api/v4/geo_nodes/1/repair" + } } ``` @@ -84,16 +107,18 @@ _This can only be run against a primary Geo node._ PUT /geo_nodes/:id ``` -| Attribute | Type | Required | Description | -|----------------------|---------|-----------|---------------------------------------------------------------------------| -| `id` | integer | yes | The ID of the Geo node. | -| `enabled` | boolean | no | Flag indicating if the Geo node is enabled. | -| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in gitlab.rb, otherwise it must match `external_url`. | -| `url` | string | yes | The user-facing URL of the Geo node. | -| `internal_url` | string | no | The URL defined on the primary node that secondary nodes should use to contact it. Returns `url` if not set.| -| `files_max_capacity` | integer | no | Control the maximum concurrency of LFS/attachment backfill for this secondary node. | -| `repos_max_capacity` | integer | no | Control the maximum concurrency of repository backfill for this secondary node. | -| `verification_max_capacity` | integer | no | Control the maximum concurrency of verification for this node. | +| Attribute | Type | Required | Description | +|-----------------------------|---------|-----------|---------------------------------------------------------------------------| +| `id` | integer | yes | The ID of the Geo node. | +| `enabled` | boolean | no | Flag indicating if the Geo node is enabled. | +| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in `gitlab.rb`, otherwise it must match `external_url`. | +| `url` | string | yes | The user-facing URL of the Geo node. | +| `internal_url` | string | no | The URL defined on the primary node that secondary nodes should use to contact it. Returns `url` if not set.| +| `files_max_capacity` | integer | no | Control the maximum concurrency of LFS/attachment backfill for this secondary node. | +| `repos_max_capacity` | integer | no | Control the maximum concurrency of repository backfill for this secondary node. | +| `verification_max_capacity` | integer | no | Control the maximum concurrency of verification for this node. | +| `container_repositories_max_capacity` | integer | no | Control the maximum concurrency of container repository sync for this node. | +| `sync_object_storage` | boolean | no | Flag indicating if the secondary Geo node will replicate blobs in Object Storage. | Example response: @@ -108,8 +133,17 @@ Example response: "current": true, "files_max_capacity": 10, "repos_max_capacity": 25, + "container_repositories_max_capacity": 10, "verification_max_capacity": 100, - "clone_protocol": "http" + "sync_object_storage": true, + "clone_protocol": "http", + "web_edit_url": "https://primary.example.com/admin/geo/nodes/2/edit", + "web_geo_projects_url": "https://secondary.example.com/admin/geo/projects", + "_links": { + "self":"https://primary.example.com/api/v4/geo_nodes/2", + "status":"https://primary.example.com/api/v4/geo_nodes/2/status", + "repair":"https://primary.example.com/api/v4/geo_nodes/2/repair" + } } ``` @@ -151,8 +185,15 @@ Example response: "current": true, "files_max_capacity": 10, "repos_max_capacity": 25, + "container_repositories_max_capacity": 10, "verification_max_capacity": 100, - "clone_protocol": "http" + "clone_protocol": "http", + "web_edit_url": "https://primary.example.com/admin/geo/nodes/1/edit", + "_links": { + "self": "https://primary.example.com/api/v4/geo_nodes/1", + "status":"https://primary.example.com/api/v4/geo_nodes/1/status", + "repair":"https://primary.example.com/api/v4/geo_nodes/1/repair" + } } ``` @@ -163,7 +204,7 @@ GET /geo_nodes/status ``` ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/status +curl --header "PRIVATE-TOKEN: <your_access_token>" https://primary.example.com/api/v4/geo_nodes/status ``` Example response: @@ -192,6 +233,10 @@ Example response: "job_artifacts_failed_count": nil, "job_artifacts_synced_missing_on_primary_count": 0, "job_artifacts_synced_in_percentage": "0.00%", + "container_repositories_count": 3, + "container_repositories_synced_count": nil, + "container_repositories_failed_count": nil, + "container_repositories_synced_in_percentage": "0.00%", "projects_count": 41, "repositories_failed_count": nil, "repositories_synced_count": nil, @@ -255,6 +300,10 @@ Example response: "job_artifacts_failed_count": 1, "job_artifacts_synced_missing_on_primary_count": 0, "job_artifacts_synced_in_percentage": "50.00%", + "container_repositories_count": 3, + "container_repositories_synced_count": nil, + "container_repositories_failed_count": nil, + "container_repositories_synced_in_percentage": "0.00%", "projects_count": 41, "repositories_failed_count": 1, "repositories_synced_count": 40, @@ -306,7 +355,7 @@ GET /geo_nodes/:id/status ``` ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/2/status +curl --header "PRIVATE-TOKEN: <your_access_token>" https://primary.example.com/api/v4/geo_nodes/2/status ``` Example response: @@ -334,6 +383,10 @@ Example response: "job_artifacts_failed_count": 1, "job_artifacts_synced_missing_on_primary_count": 0, "job_artifacts_synced_in_percentage": "50.00%", + "container_repositories_count": 3, + "container_repositories_synced_count": nil, + "container_repositories_failed_count": nil, + "container_repositories_synced_in_percentage": "0.00%", "projects_count": 41, "repositories_failed_count": 1, "repositories_synced_count": 40, @@ -376,7 +429,7 @@ GET /geo_nodes/current/failures This endpoint uses [Pagination](README.md#pagination). ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/current/failures +curl --header "PRIVATE-TOKEN: <your_access_token>" https://primary.example.com/api/v4/geo_nodes/current/failures ``` Example response: diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index db073321808..bdc7c1959d2 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -1,6 +1,8 @@ # GraphQL API -> [Introduced][ce-19008] in GitLab 11.0. +> - [Introduced][ce-19008] in GitLab 11.0 (enabled by feature flag `graphql`). +> - [Always enabled](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30444) + in GitLab 12.1. [GraphQL](https://graphql.org/) is a query language for APIs that allows clients to request exactly the data they need, making it @@ -29,8 +31,6 @@ There are no plans to deprecate the REST API. To reduce the technical burden of supporting two APIs in parallel, they should share implementations as much as possible. -As of the 12.1 release, GraphQL is always enabled. - ## Available queries A first iteration of a GraphQL API includes the following queries @@ -47,6 +47,12 @@ info about multiplexed queries is also available for [graphql-ruby](https://graphql-ruby.org/queries/multiplex.html) the library GitLab uses on the backend. +## Reference + +GitLab's GraphQL reference [is available](reference/index.md). + +It is automatically generated from GitLab's GraphQL schema and embedded in a Markdown file. + ## GraphiQL The API can be explored by using the GraphiQL IDE, it is available on your diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md new file mode 100644 index 00000000000..d99a4c37d72 --- /dev/null +++ b/doc/api/graphql/reference/index.md @@ -0,0 +1,519 @@ +<!--- + This documentation is auto generated by a script. + + Please do not edit this file directly, check compile_docs task on lib/tasks/gitlab/graphql.rake. +---> + +# GraphQL API Resources + +This documentation is self-generated based on GitLab current GraphQL schema. + +The API can be explored interactively using the [GraphiQL IDE](../index.md#graphiql). + +## Objects + +### AddAwardEmojiPayload + +| Name | Type | Description | +| --- | ---- | ---------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Reasons why the mutation failed. | +| `awardEmoji` | AwardEmoji | The award emoji after mutation | + +### AwardEmoji + +| Name | Type | Description | +| --- | ---- | ---------- | +| `name` | String! | The emoji name | +| `description` | String! | The emoji description | +| `unicode` | String! | The emoji in unicode | +| `emoji` | String! | The emoji as an icon | +| `unicodeVersion` | String! | The unicode version for this emoji | +| `user` | User! | The user who awarded the emoji | + +### Blob + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `name` | String! | | +| `type` | EntryType! | | +| `path` | String! | | +| `flatPath` | String! | | +| `webUrl` | String | | +| `lfsOid` | String | | + +### Commit + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `sha` | String! | | +| `title` | String | | +| `description` | String | | +| `message` | String | | +| `authoredDate` | Time | | +| `webUrl` | String! | | +| `author` | User | | +| `latestPipeline` | Pipeline | Latest pipeline for this commit | + +### DetailedStatus + +| Name | Type | Description | +| --- | ---- | ---------- | +| `group` | String! | | +| `icon` | String! | | +| `favicon` | String! | | +| `detailsPath` | String! | | +| `hasDetails` | Boolean! | | +| `label` | String! | | +| `text` | String! | | +| `tooltip` | String! | | + +### DiffPosition + +| Name | Type | Description | +| --- | ---- | ---------- | +| `headSha` | String! | The sha of the head at the time the comment was made | +| `baseSha` | String | The merge base of the branch the comment was made on | +| `startSha` | String! | The sha of the branch being compared against | +| `filePath` | String! | The path of the file that was changed | +| `oldPath` | String | The path of the file on the start sha. | +| `newPath` | String | The path of the file on the head sha. | +| `positionType` | DiffPositionType! | | +| `oldLine` | Int | The line on start sha that was changed | +| `newLine` | Int | The line on head sha that was changed | +| `x` | Int | The X postion on which the comment was made | +| `y` | Int | The Y position on which the comment was made | +| `width` | Int | The total width of the image | +| `height` | Int | The total height of the image | + +### Discussion + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `createdAt` | Time! | | + +### Group + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `name` | String! | | +| `path` | String! | | +| `fullName` | String! | | +| `fullPath` | ID! | | +| `description` | String | | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | +| `visibility` | String | | +| `lfsEnabled` | Boolean | | +| `requestAccessEnabled` | Boolean | | +| `rootStorageStatistics` | RootStorageStatistics | The aggregated storage statistics. Only available if the namespace has no parent | +| `userPermissions` | GroupPermissions! | Permissions for the current user on the resource | +| `webUrl` | String! | | +| `avatarUrl` | String | | +| `parent` | Group | | + +### GroupPermissions + +| Name | Type | Description | +| --- | ---- | ---------- | +| `readGroup` | Boolean! | Whether or not a user can perform `read_group` on this resource | + +### Issue + +| Name | Type | Description | +| --- | ---- | ---------- | +| `userPermissions` | IssuePermissions! | Permissions for the current user on the resource | +| `iid` | ID! | | +| `title` | String! | | +| `titleHtml` | String | The GitLab Flavored Markdown rendering of `title` | +| `description` | String | | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | +| `state` | IssueState! | | +| `reference` | String! | | +| `author` | User! | | +| `milestone` | Milestone | | +| `dueDate` | Time | | +| `confidential` | Boolean! | | +| `discussionLocked` | Boolean! | | +| `upvotes` | Int! | | +| `downvotes` | Int! | | +| `userNotesCount` | Int! | | +| `webPath` | String! | | +| `webUrl` | String! | | +| `relativePosition` | Int | | +| `closedAt` | Time | | +| `createdAt` | Time! | | +| `updatedAt` | Time! | | +| `taskCompletionStatus` | TaskCompletionStatus! | | + +### IssuePermissions + +| Name | Type | Description | +| --- | ---- | ---------- | +| `readIssue` | Boolean! | Whether or not a user can perform `read_issue` on this resource | +| `adminIssue` | Boolean! | Whether or not a user can perform `admin_issue` on this resource | +| `updateIssue` | Boolean! | Whether or not a user can perform `update_issue` on this resource | +| `createNote` | Boolean! | Whether or not a user can perform `create_note` on this resource | +| `reopenIssue` | Boolean! | Whether or not a user can perform `reopen_issue` on this resource | + +### Label + +| Name | Type | Description | +| --- | ---- | ---------- | +| `description` | String | | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | +| `title` | String! | | +| `color` | String! | | +| `textColor` | String! | | + +### MergeRequest + +| Name | Type | Description | +| --- | ---- | ---------- | +| `userPermissions` | MergeRequestPermissions! | Permissions for the current user on the resource | +| `id` | ID! | | +| `iid` | String! | | +| `title` | String! | | +| `titleHtml` | String | The GitLab Flavored Markdown rendering of `title` | +| `description` | String | | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | +| `state` | MergeRequestState! | | +| `createdAt` | Time! | | +| `updatedAt` | Time! | | +| `sourceProject` | Project | | +| `targetProject` | Project! | | +| `project` | Project! | | +| `projectId` | Int! | | +| `sourceProjectId` | Int | | +| `targetProjectId` | Int! | | +| `sourceBranch` | String! | | +| `targetBranch` | String! | | +| `workInProgress` | Boolean! | | +| `mergeWhenPipelineSucceeds` | Boolean | | +| `diffHeadSha` | String | | +| `mergeCommitSha` | String | | +| `userNotesCount` | Int | | +| `shouldRemoveSourceBranch` | Boolean | | +| `forceRemoveSourceBranch` | Boolean | | +| `mergeStatus` | String | | +| `inProgressMergeCommitSha` | String | | +| `mergeError` | String | | +| `allowCollaboration` | Boolean | | +| `shouldBeRebased` | Boolean! | | +| `rebaseCommitSha` | String | | +| `rebaseInProgress` | Boolean! | | +| `mergeCommitMessage` | String | | +| `defaultMergeCommitMessage` | String | | +| `mergeOngoing` | Boolean! | | +| `sourceBranchExists` | Boolean! | | +| `mergeableDiscussionsState` | Boolean | | +| `webUrl` | String | | +| `upvotes` | Int! | | +| `downvotes` | Int! | | +| `subscribed` | Boolean! | | +| `headPipeline` | Pipeline | | +| `taskCompletionStatus` | TaskCompletionStatus! | | + +### MergeRequestPermissions + +| Name | Type | Description | +| --- | ---- | ---------- | +| `readMergeRequest` | Boolean! | Whether or not a user can perform `read_merge_request` on this resource | +| `adminMergeRequest` | Boolean! | Whether or not a user can perform `admin_merge_request` on this resource | +| `updateMergeRequest` | Boolean! | Whether or not a user can perform `update_merge_request` on this resource | +| `createNote` | Boolean! | Whether or not a user can perform `create_note` on this resource | +| `pushToSourceBranch` | Boolean! | Whether or not a user can perform `push_to_source_branch` on this resource | +| `removeSourceBranch` | Boolean! | Whether or not a user can perform `remove_source_branch` on this resource | +| `cherryPickOnCurrentMergeRequest` | Boolean! | Whether or not a user can perform `cherry_pick_on_current_merge_request` on this resource | +| `revertOnCurrentMergeRequest` | Boolean! | Whether or not a user can perform `revert_on_current_merge_request` on this resource | + +### MergeRequestSetWipPayload + +| Name | Type | Description | +| --- | ---- | ---------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Reasons why the mutation failed. | +| `mergeRequest` | MergeRequest | The merge request after mutation | + +### Metadata + +| Name | Type | Description | +| --- | ---- | ---------- | +| `version` | String! | | +| `revision` | String! | | + +### Milestone + +| Name | Type | Description | +| --- | ---- | ---------- | +| `description` | String | | +| `title` | String! | | +| `state` | String! | | +| `dueDate` | Time | | +| `startDate` | Time | | +| `createdAt` | Time! | | +| `updatedAt` | Time! | | + +### Namespace + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `name` | String! | | +| `path` | String! | | +| `fullName` | String! | | +| `fullPath` | ID! | | +| `description` | String | | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | +| `visibility` | String | | +| `lfsEnabled` | Boolean | | +| `requestAccessEnabled` | Boolean | | + +### Note + +| Name | Type | Description | +| --- | ---- | ---------- | +| `userPermissions` | NotePermissions! | Permissions for the current user on the resource | +| `id` | ID! | | +| `project` | Project | The project this note is associated to | +| `author` | User! | The user who wrote this note | +| `resolvedBy` | User | The user that resolved the discussion | +| `system` | Boolean! | Whether or not this note was created by the system or by a user | +| `body` | String! | The content note itself | +| `bodyHtml` | String | The GitLab Flavored Markdown rendering of `note` | +| `createdAt` | Time! | | +| `updatedAt` | Time! | | +| `discussion` | Discussion | The discussion this note is a part of | +| `resolvable` | Boolean! | | +| `resolvedAt` | Time | The time the discussion was resolved | +| `position` | DiffPosition | The position of this note on a diff | + +### NotePermissions + +| Name | Type | Description | +| --- | ---- | ---------- | +| `readNote` | Boolean! | Whether or not a user can perform `read_note` on this resource | +| `createNote` | Boolean! | Whether or not a user can perform `create_note` on this resource | +| `adminNote` | Boolean! | Whether or not a user can perform `admin_note` on this resource | +| `resolveNote` | Boolean! | Whether or not a user can perform `resolve_note` on this resource | +| `awardEmoji` | Boolean! | Whether or not a user can perform `award_emoji` on this resource | + +### PageInfo + +| Name | Type | Description | +| --- | ---- | ---------- | +| `hasNextPage` | Boolean! | When paginating forwards, are there more items? | +| `hasPreviousPage` | Boolean! | When paginating backwards, are there more items? | +| `startCursor` | String | When paginating backwards, the cursor to continue. | +| `endCursor` | String | When paginating forwards, the cursor to continue. | + +### Pipeline + +| Name | Type | Description | +| --- | ---- | ---------- | +| `userPermissions` | PipelinePermissions! | Permissions for the current user on the resource | +| `id` | ID! | | +| `iid` | String! | | +| `sha` | String! | | +| `beforeSha` | String | | +| `status` | PipelineStatusEnum! | | +| `detailedStatus` | DetailedStatus! | | +| `duration` | Int | Duration of the pipeline in seconds | +| `coverage` | Float | Coverage percentage | +| `createdAt` | Time! | | +| `updatedAt` | Time! | | +| `startedAt` | Time | | +| `finishedAt` | Time | | +| `committedAt` | Time | | + +### PipelinePermissions + +| Name | Type | Description | +| --- | ---- | ---------- | +| `updatePipeline` | Boolean! | Whether or not a user can perform `update_pipeline` on this resource | +| `adminPipeline` | Boolean! | Whether or not a user can perform `admin_pipeline` on this resource | +| `destroyPipeline` | Boolean! | Whether or not a user can perform `destroy_pipeline` on this resource | + +### Project + +| Name | Type | Description | +| --- | ---- | ---------- | +| `userPermissions` | ProjectPermissions! | Permissions for the current user on the resource | +| `id` | ID! | | +| `fullPath` | ID! | | +| `path` | String! | | +| `nameWithNamespace` | String! | | +| `name` | String! | | +| `description` | String | | +| `descriptionHtml` | String | The GitLab Flavored Markdown rendering of `description` | +| `tagList` | String | | +| `sshUrlToRepo` | String | | +| `httpUrlToRepo` | String | | +| `webUrl` | String | | +| `starCount` | Int! | | +| `forksCount` | Int! | | +| `createdAt` | Time | | +| `lastActivityAt` | Time | | +| `archived` | Boolean | | +| `visibility` | String | | +| `containerRegistryEnabled` | Boolean | | +| `sharedRunnersEnabled` | Boolean | | +| `lfsEnabled` | Boolean | | +| `mergeRequestsFfOnlyEnabled` | Boolean | | +| `avatarUrl` | String | | +| `issuesEnabled` | Boolean | | +| `mergeRequestsEnabled` | Boolean | | +| `wikiEnabled` | Boolean | | +| `snippetsEnabled` | Boolean | | +| `jobsEnabled` | Boolean | | +| `publicJobs` | Boolean | | +| `openIssuesCount` | Int | | +| `importStatus` | String | | +| `onlyAllowMergeIfPipelineSucceeds` | Boolean | | +| `requestAccessEnabled` | Boolean | | +| `onlyAllowMergeIfAllDiscussionsAreResolved` | Boolean | | +| `printingMergeRequestLinkEnabled` | Boolean | | +| `namespace` | Namespace | | +| `group` | Group | | +| `statistics` | ProjectStatistics | | +| `repository` | Repository | | +| `mergeRequest` | MergeRequest | | +| `issue` | Issue | | + +### ProjectPermissions + +| Name | Type | Description | +| --- | ---- | ---------- | +| `changeNamespace` | Boolean! | Whether or not a user can perform `change_namespace` on this resource | +| `changeVisibilityLevel` | Boolean! | Whether or not a user can perform `change_visibility_level` on this resource | +| `renameProject` | Boolean! | Whether or not a user can perform `rename_project` on this resource | +| `removeProject` | Boolean! | Whether or not a user can perform `remove_project` on this resource | +| `archiveProject` | Boolean! | Whether or not a user can perform `archive_project` on this resource | +| `removeForkProject` | Boolean! | Whether or not a user can perform `remove_fork_project` on this resource | +| `removePages` | Boolean! | Whether or not a user can perform `remove_pages` on this resource | +| `readProject` | Boolean! | Whether or not a user can perform `read_project` on this resource | +| `createMergeRequestIn` | Boolean! | Whether or not a user can perform `create_merge_request_in` on this resource | +| `readWiki` | Boolean! | Whether or not a user can perform `read_wiki` on this resource | +| `readProjectMember` | Boolean! | Whether or not a user can perform `read_project_member` on this resource | +| `createIssue` | Boolean! | Whether or not a user can perform `create_issue` on this resource | +| `uploadFile` | Boolean! | Whether or not a user can perform `upload_file` on this resource | +| `readCycleAnalytics` | Boolean! | Whether or not a user can perform `read_cycle_analytics` on this resource | +| `downloadCode` | Boolean! | Whether or not a user can perform `download_code` on this resource | +| `downloadWikiCode` | Boolean! | Whether or not a user can perform `download_wiki_code` on this resource | +| `forkProject` | Boolean! | Whether or not a user can perform `fork_project` on this resource | +| `createProjectSnippet` | Boolean! | Whether or not a user can perform `create_project_snippet` on this resource | +| `readCommitStatus` | Boolean! | Whether or not a user can perform `read_commit_status` on this resource | +| `requestAccess` | Boolean! | Whether or not a user can perform `request_access` on this resource | +| `createPipeline` | Boolean! | Whether or not a user can perform `create_pipeline` on this resource | +| `createPipelineSchedule` | Boolean! | Whether or not a user can perform `create_pipeline_schedule` on this resource | +| `createMergeRequestFrom` | Boolean! | Whether or not a user can perform `create_merge_request_from` on this resource | +| `createWiki` | Boolean! | Whether or not a user can perform `create_wiki` on this resource | +| `pushCode` | Boolean! | Whether or not a user can perform `push_code` on this resource | +| `createDeployment` | Boolean! | Whether or not a user can perform `create_deployment` on this resource | +| `pushToDeleteProtectedBranch` | Boolean! | Whether or not a user can perform `push_to_delete_protected_branch` on this resource | +| `adminWiki` | Boolean! | Whether or not a user can perform `admin_wiki` on this resource | +| `adminProject` | Boolean! | Whether or not a user can perform `admin_project` on this resource | +| `updatePages` | Boolean! | Whether or not a user can perform `update_pages` on this resource | +| `adminRemoteMirror` | Boolean! | Whether or not a user can perform `admin_remote_mirror` on this resource | +| `createLabel` | Boolean! | Whether or not a user can perform `create_label` on this resource | +| `updateWiki` | Boolean! | Whether or not a user can perform `update_wiki` on this resource | +| `destroyWiki` | Boolean! | Whether or not a user can perform `destroy_wiki` on this resource | +| `createPages` | Boolean! | Whether or not a user can perform `create_pages` on this resource | +| `destroyPages` | Boolean! | Whether or not a user can perform `destroy_pages` on this resource | +| `readPagesContent` | Boolean! | Whether or not a user can perform `read_pages_content` on this resource | + +### ProjectStatistics + +| Name | Type | Description | +| --- | ---- | ---------- | +| `commitCount` | Int! | | +| `storageSize` | Int! | | +| `repositorySize` | Int! | | +| `lfsObjectsSize` | Int! | | +| `buildArtifactsSize` | Int! | | +| `packagesSize` | Int! | | +| `wikiSize` | Int | | + +### RemoveAwardEmojiPayload + +| Name | Type | Description | +| --- | ---- | ---------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Reasons why the mutation failed. | +| `awardEmoji` | AwardEmoji | The award emoji after mutation | + +### Repository + +| Name | Type | Description | +| --- | ---- | ---------- | +| `rootRef` | String | | +| `empty` | Boolean! | | +| `exists` | Boolean! | | +| `tree` | Tree | | + +### RootStorageStatistics + +| Name | Type | Description | +| --- | ---- | ---------- | +| `storageSize` | Int! | The total storage in Bytes | +| `repositorySize` | Int! | The git repository size in Bytes | +| `lfsObjectsSize` | Int! | The LFS objects size in Bytes | +| `buildArtifactsSize` | Int! | The CI artifacts size in Bytes | +| `packagesSize` | Int! | The packages size in Bytes | +| `wikiSize` | Int! | The wiki size in Bytes | + +### Submodule + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `name` | String! | | +| `type` | EntryType! | | +| `path` | String! | | +| `flatPath` | String! | | + +### TaskCompletionStatus + +| Name | Type | Description | +| --- | ---- | ---------- | +| `count` | Int! | | +| `completedCount` | Int! | | + +### ToggleAwardEmojiPayload + +| Name | Type | Description | +| --- | ---- | ---------- | +| `clientMutationId` | String | A unique identifier for the client performing the mutation. | +| `errors` | String! => Array | Reasons why the mutation failed. | +| `awardEmoji` | AwardEmoji | The award emoji after mutation | +| `toggledOn` | Boolean! | True when the emoji was awarded, false when it was removed | + +### Tree + +| Name | Type | Description | +| --- | ---- | ---------- | +| `lastCommit` | Commit | | + +### TreeEntry + +| Name | Type | Description | +| --- | ---- | ---------- | +| `id` | ID! | | +| `name` | String! | | +| `type` | EntryType! | | +| `path` | String! | | +| `flatPath` | String! | | +| `webUrl` | String | | + +### User + +| Name | Type | Description | +| --- | ---- | ---------- | +| `name` | String! | | +| `username` | String! | | +| `avatarUrl` | String! | | +| `webUrl` | String! | | + diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md index 4d10f83720b..99b522a7ae9 100644 --- a/doc/api/group_boards.md +++ b/doc/api/group_boards.md @@ -536,7 +536,7 @@ Example response: ## Delete a group issue board list -Only for admins and group owners. Soft deletes the board list in question. +Only for admins and group owners. Deletes the board list in question. ``` DELETE /groups/:id/boards/:board_id/lists/:list_id diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md new file mode 100644 index 00000000000..29e58d9279a --- /dev/null +++ b/doc/api/group_clusters.md @@ -0,0 +1,280 @@ +# Group clusters API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30213) +in GitLab 12.1. + +NOTE: **Note:** +User will need at least maintainer access for the group to use these endpoints. + +## List group clusters + +Returns a list of group clusters. + +``` +GET /groups/:id/clusters +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | + +Example request: + +```bash +curl --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/groups/26/clusters +``` + +Example response: + +```json +[ + { + "id":18, + "name":"cluster-1", + "domain":"example.com", + "created_at":"2019-01-02T20:18:12.563Z", + "provider_type":"user", + "platform_type":"kubernetes", + "environment_scope":"*", + "cluster_type":"group_type", + "user": + { + "id":1, + "name":"Administrator", + "username":"root", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/4249f4df72b..", + "web_url":"https://gitlab.example.com/root" + }, + "platform_kubernetes": + { + "api_url":"https://104.197.68.152", + "authorization_type":"rbac", + "ca_cert":"-----BEGIN CERTIFICATE-----\r\nhFiK1L61owwDQYJKoZIhvcNAQELBQAw\r\nLzEtMCsGA1UEAxMkZDA1YzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM4ZDBj\r\nMB4XDTE4MTIyNzIwMDM1MVoXDTIzMTIyNjIxMDM1MVowLzEtMCsGA1UEAxMkZDA1\r\nYzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM.......-----END CERTIFICATE-----" + } + }, + { + "id":19, + "name":"cluster-2", + ... + } +] +``` + +## Get a single group cluster + +Gets a single group cluster. + +``` +GET /groups/:id/clusters/:cluster_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `cluster_id` | integer | yes | The ID of the cluster | + +Example request: + +```bash +curl --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/groups/26/clusters/18 +``` + +Example response: + +```json +{ + "id":18, + "name":"cluster-1", + "domain":"example.com", + "created_at":"2019-01-02T20:18:12.563Z", + "provider_type":"user", + "platform_type":"kubernetes", + "environment_scope":"*", + "cluster_type":"group_type", + "user": + { + "id":1, + "name":"Administrator", + "username":"root", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/4249f4df72b..", + "web_url":"https://gitlab.example.com/root" + }, + "platform_kubernetes": + { + "api_url":"https://104.197.68.152", + "authorization_type":"rbac", + "ca_cert":"-----BEGIN CERTIFICATE-----\r\nhFiK1L61owwDQYJKoZIhvcNAQELBQAw\r\nLzEtMCsGA1UEAxMkZDA1YzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM4ZDBj\r\nMB4XDTE4MTIyNzIwMDM1MVoXDTIzMTIyNjIxMDM1MVowLzEtMCsGA1UEAxMkZDA1\r\nYzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM.......-----END CERTIFICATE-----" + }, + "group": + { + "id":26, + "name":"group-with-clusters-api", + "web_url":"https://gitlab.example.com/group-with-clusters-api" + } +} +``` + +## Add existing cluster to group + +Adds an existing Kubernetes cluster to the group. + +``` +POST /groups/:id/clusters/user +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `name` | String | yes | The name of the cluster | +| `domain` | String | no | The [base domain](../user/group/clusters/index.md#base-domain) of the cluster | +| `enabled` | Boolean | no | Determines if cluster is active or not, defaults to true | +| `managed` | Boolean | no | Determines if GitLab will manage namespaces and service accounts for this cluster, defaults to true | +| `platform_kubernetes_attributes[api_url]` | String | yes | The URL to access the Kubernetes API | +| `platform_kubernetes_attributes[token]` | String | yes | The token to authenticate against Kubernetes | +| `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | +| `platform_kubernetes_attributes[authorization_type]` | String | no | The cluster authorization type: `rbac`, `abac` or `unknown_authorization`. Defaults to `rbac`. | +| `environment_scope` | String | no | The associated environment to the cluster. Defaults to `*` **(PREMIUM)** | + +Example request: + +```bash +curl --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/groups/26/clusters/user \ +-H "Accept: application/json" \ +-H "Content-Type:application/json" \ +--request POST --data '{"name":"cluster-5", "platform_kubernetes_attributes":{"api_url":"https://35.111.51.20","token":"12345","ca_cert":"-----BEGIN CERTIFICATE-----\r\nhFiK1L61owwDQYJKoZIhvcNAQELBQAw\r\nLzEtMCsGA1UEAxMkZDA1YzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM4ZDBj\r\nMB4XDTE4MTIyNzIwMDM1MVoXDTIzMTIyNjIxMDM1MVowLzEtMCsGA1UEAxMkZDA1\r\nYzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM.......-----END CERTIFICATE-----"}}' +``` + +Example response: + +```json +{ + "id":24, + "name":"cluster-5", + "created_at":"2019-01-03T21:53:40.610Z", + "provider_type":"user", + "platform_type":"kubernetes", + "environment_scope":"*", + "cluster_type":"group_type", + "user": + { + "id":1, + "name":"Administrator", + "username":"root", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/4249f4df72b..", + "web_url":"https://gitlab.example.com/root" + }, + "platform_kubernetes": + { + "api_url":"https://35.111.51.20", + "authorization_type":"rbac", + "ca_cert":"-----BEGIN CERTIFICATE-----\r\nhFiK1L61owwDQYJKoZIhvcNAQELBQAw\r\nLzEtMCsGA1UEAxMkZDA1YzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM4ZDBj\r\nMB4XDTE4MTIyNzIwMDM1MVoXDTIzMTIyNjIxMDM1MVowLzEtMCsGA1UEAxMkZDA1\r\nYzQ1YjctNzdiMS00NDY0LThjNmEtMTQ0ZDJkZjM.......-----END CERTIFICATE-----" + }, + "group": + { + "id":26, + "name":"group-with-clusters-api", + "web_url":"https://gitlab.example.com/root/group-with-clusters-api" + } +} +``` + +## Edit group cluster + +Updates an existing group cluster. + +``` +PUT /groups/:id/clusters/:cluster_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `cluster_id` | integer | yes | The ID of the cluster | +| `name` | String | no | The name of the cluster | +| `domain` | String | no | The [base domain](../user/group/clusters/index.md#base-domain) of the cluster | +| `platform_kubernetes_attributes[api_url]` | String | no | The URL to access the Kubernetes API | +| `platform_kubernetes_attributes[token]` | String | no | The token to authenticate against Kubernetes | +| `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | +| `environment_scope` | String | no | The associated environment to the cluster **(PREMIUM)** | + +NOTE: **Note:** +`name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added +through the ["Add existing Kubernetes cluster"](../user/project/clusters/index.md#add-existing-kubernetes-cluster) option or +through the ["Add existing cluster to group"](#add-existing-cluster-to-group) endpoint. + +Example request: + +```bash +curl --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/groups/26/clusters/24 \ +-H "Content-Type:application/json" \ +--request PUT --data '{"name":"new-cluster-name","domain":"new-domain.com","api_url":"https://new-api-url.com"}' +``` + +Example response: + +```json +{ + "id":24, + "name":"new-cluster-name", + "domain":"new-domain.com", + "created_at":"2019-01-03T21:53:40.610Z", + "provider_type":"user", + "platform_type":"kubernetes", + "environment_scope":"*", + "cluster_type":"group_type", + "user": + { + "id":1, + "name":"Administrator", + "username":"root", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/4249f4df72b..", + "web_url":"https://gitlab.example.com/root" + }, + "platform_kubernetes": + { + "api_url":"https://new-api-url.com", + "authorization_type":"rbac", + "ca_cert":null + }, + "group": + { + "id":26, + "name":"group-with-clusters-api", + "web_url":"https://gitlab.example.com/group-with-clusters-api" + } +} + +``` + +## Delete group cluster + +Deletes an existing group cluster. + +``` +DELETE /groups/:id/clusters/:cluster_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `cluster_id` | integer | yes | The ID of the cluster | + +Example request: + +```bash +curl --request DELETE --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/groups/26/clusters/23' +``` diff --git a/doc/api/group_labels.md b/doc/api/group_labels.md index 3d4b099b49e..e2ba0dea642 100644 --- a/doc/api/group_labels.md +++ b/doc/api/group_labels.md @@ -12,12 +12,13 @@ Get all labels for a given group. GET /groups/:id/labels ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user. | +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `with_counts` | boolean | no | Whether or not to include issue and merge request counts. Defaults to `false`. _([Introduced in GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31543))_ | ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/labels +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/labels?with_counts=true ``` Example response: diff --git a/doc/api/groups.md b/doc/api/groups.md index d05e4b29fef..0d500f783aa 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -634,11 +634,13 @@ Parameters: By default, groups only get 20 namespaces at a time because the API results are paginated. To get more (up to 100), pass the following as an argument to the API call: + ``` /groups?per_page=100 ``` And to switch pages add: + ``` /groups?per_page=100&page=2 ``` diff --git a/doc/api/issues.md b/doc/api/issues.md index 23126a05b66..cadc9291489 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -49,7 +49,7 @@ GET /issues?confidential=true | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `iids[]` | integer array | no | Return only the issues having the given `iid` | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | | `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | @@ -136,7 +136,6 @@ Example response: "award_emoji":"http://example.com/api/v4/projects/1/issues/76/award_emoji", "project":"http://example.com/api/v4/projects/1" }, - "subscribed": false, "task_completion_status":{ "count":0, "completed_count":0 @@ -199,7 +198,7 @@ GET /groups/:id/issues?confidential=true | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search group issues against their `title` and `description` | | `created_after` | datetime | no | Return issues created on or after the given time | @@ -285,7 +284,6 @@ Example response: "award_emoji":"http://example.com/api/v4/projects/4/issues/41/award_emoji", "project":"http://example.com/api/v4/projects/4" }, - "subscribed": false, "task_completion_status":{ "count":0, "completed_count":0 @@ -348,7 +346,7 @@ GET /projects/:id/issues?confidential=true | `assignee_username` | string array | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `weight` **(STARTER)** | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | -| `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `order_by` | string | no | Return issues ordered by `created_at`, `updated_at`, `priority`, `due_date`, `relative_position`, `label_priority`, `milestone_due`, `popularity`, `weight` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search project issues against their `title` and `description` | | `created_after` | datetime | no | Return issues created on or after the given time | @@ -357,7 +355,6 @@ GET /projects/:id/issues?confidential=true | `updated_before` | datetime | no | Return issues updated on or before the given time | | `confidential` | Boolean | no | Filter confidential or public issues. | - ```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/issues ``` @@ -442,7 +439,6 @@ Example response: "award_emoji":"http://example.com/api/v4/projects/4/issues/41/award_emoji", "project":"http://example.com/api/v4/projects/4" }, - "subscribed": false, "task_completion_status":{ "count":0, "completed_count":0 @@ -791,7 +787,7 @@ the `weight` parameter: ## Delete an issue -Only for admins and project owners. Soft deletes the issue in question. +Only for admins and project owners. Deletes the issue in question. ``` DELETE /projects/:id/issues/:issue_iid diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 1add5f432ac..2a1c1b5f6f3 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -409,10 +409,10 @@ Possible response status codes: > - The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] > in [GitLab Premium][ee] 9.5. -Download the artifacts zipped archive from the given reference name and job, -provided the job finished successfully. This is the same as -[getting the job's artifacts](#get-job-artifacts), but by defining the job's -name instead of its ID. +Download the artifacts zipped archive from the latest successful pipeline for +the given reference name and job, provided the job finished successfully. This +is the same as [getting the job's artifacts](#get-job-artifacts), but by +defining the job's name instead of its ID. ``` GET /projects/:id/jobs/artifacts/:ref_name/download?job=name @@ -491,7 +491,7 @@ Parameters Example request: ```sh -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/5/artifacts/some/release/file.pdf" +curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/5/artifacts/some/release/file.pdf" ``` Possible response status codes: @@ -506,9 +506,9 @@ Possible response status codes: > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23538) in GitLab 11.5. -Download a single artifact file from a specific tag or branch from within the -job's artifacts archive. The file is extracted from the archive and streamed to -the client. +Download a single artifact file for a specific job of the latest successful +pipeline for the given reference name from within the job's artifacts archive. +The file is extracted from the archive and streamed to the client. ``` GET /projects/:id/jobs/artifacts/:ref_name/raw/*artifact_path?job=name @@ -526,7 +526,7 @@ Parameters: Example request: ```sh -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/raw/some/release/file.pdf?job=pdf" +curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/raw/some/release/file.pdf?job=pdf" ``` Possible response status codes: @@ -551,7 +551,7 @@ GET /projects/:id/jobs/:job_id/trace | job_id | integer | yes | ID of a job. | ```sh -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8/trace" +curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8/trace" ``` Possible response status codes: diff --git a/doc/api/labels.md b/doc/api/labels.md index 9d10d383bf9..9692cc8b710 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -8,12 +8,13 @@ Get all labels for a given project. GET /projects/:id/labels ``` -| Attribute | Type | Required | Description | -| --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `with_counts` | boolean | no | Whether or not to include issue and merge request counts. Defaults to `false`. _([Introduced in GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31543))_ | ```bash -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/labels +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/labels?with_counts=true ``` Example response: @@ -136,8 +137,9 @@ DELETE /projects/:id/labels | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the label | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `label_id` | integer | yes (or `name`) | The id of the existing label | +| `name` | string | yes (or `label_id`) | The name of the existing label | ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels?name=bug" @@ -155,7 +157,8 @@ PUT /projects/:id/labels | Attribute | Type | Required | Description | | --------------- | ------- | --------------------------------- | ------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `name` | string | yes | The name of the existing label | +| `label_id` | integer | yes (or `name`) | The id of the existing label | +| `name` | string | yes (or `label_id`) | The name of the existing label | | `new_name` | string | yes if `color` is not provided | The new name of the label | | `color` | string | yes if `new_name` is not provided | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | | `description` | string | no | The new description of the label | @@ -183,6 +186,40 @@ Example response: } ``` +## Promote a project label to a group label + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25218) in GitLab 12.3. + +Promotes a project label to a group label. + +``` +PUT /projects/:id/labels/promote +``` + +| Attribute | Type | Required | Description | +| --------------- | ------- | --------------------------------- | ------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the existing label | + +```bash +curl --request PUT --data "name=documentation" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels/promote" +``` + +Example response: + +```json +{ + "id" : 8, + "name" : "documentation", + "color" : "#8E44AD", + "description": "Documentation", + "open_issues_count": 1, + "closed_issues_count": 0, + "open_merge_requests_count": 2, + "subscribed": false +} +``` + ## Subscribe to a label Subscribes the authenticated user to a label to receive notifications. diff --git a/doc/api/lint.md b/doc/api/lint.md index b9b49f3df27..dacd3f4c493 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -1,16 +1,16 @@ -# Validate the .gitlab-ci.yml (API) +# Validate the `.gitlab-ci.yml` (API) > [Introduced][ce-5953] in GitLab 8.12. Checks if your `.gitlab-ci.yml` file is valid. ``` -POST /lint +POST /ci/lint ``` | Attribute | Type | Required | Description | | ---------- | ------- | -------- | -------- | -| `content` | string | yes | the .gitlab-ci.yaml content| +| `content` | string | yes | the `.gitlab-ci.yaml` content| ```bash curl --header "Content-Type: application/json" https://gitlab.example.com/api/v4/ci/lint --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index c211916464a..bf83bfd7e6a 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -23,36 +23,6 @@ GET /projects/:id/approvals ```json { - "approvers": [ - { - "user": { - "id": 5, - "name": "John Doe6", - "username": "user5", - "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" - } - } - ], - "approver_groups": [ - { - "group": { - "id": 1, - "name": "group1", - "path": "group1", - "description": "", - "visibility": "public", - "lfs_enabled": false, - "avatar_url": null, - "web_url": "http://localhost/groups/group1", - "request_access_enabled": false, - "full_name": "group1", - "full_path": "group1", - "parent_id": null, - "ldap_cn": null, - "ldap_access": null - } - } - ], "approvals_before_merge": 2, "reset_approvals_on_push": true, "disable_overriding_approvers_per_merge_request": false @@ -72,30 +42,79 @@ POST /projects/:id/approvals **Parameters:** -| Attribute | Type | Required | Description | -| ------------------------------------------------ | ------- | -------- | ---------------------------------------------------------- | -| `id` | integer | yes | The ID of a project | -| `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged | -| `reset_approvals_on_push` | boolean | no | Reset approvals on a new push | -| `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | -| `merge_requests_author_approval` | boolean | no | Allow/Disallow authors be able to self approve merge requests | +| Attribute | Type | Required | Description | +| ------------------------------------------------ | ------- | -------- | --------------------------------------------------------------------------------------------------- | +| `id` | integer | yes | The ID of a project | +| `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged. Deprecated in 12.0 in favor of Approval Rules API. | +| `reset_approvals_on_push` | boolean | no | Reset approvals on a new push | +| `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | +| `merge_requests_author_approval` | boolean | no | Allow/Disallow authors from self approving merge requests; `true` means authors cannot self approve | +| `merge_requests_disable_committers_approval` | boolean | no | Allow/Disallow committers from self approving merge requests | ```json { - "approvers": [ - { - "user": { + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false, + "merge_requests_author_approval": false, + "merge_requests_disable_committers_approval": false +} +``` + +### Get project-level rules + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can request information about a project's approval rules using the following endpoint: + +``` +GET /projects/:id/approval_rules +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | + +```json +[ + { + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { "id": 5, - "name": "John Doe6", - "username": "user5", - "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" } - } - ], - "approver_groups": [ - { - "group": { - "id": 1, + ], + "approvals_required": 3, + "users": [ + { + "id": 5, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, "name": "group1", "path": "group1", "description": "", @@ -110,17 +129,187 @@ POST /projects/:id/approvals "ldap_cn": null, "ldap_access": null } + ], + "contains_hidden_groups": false + } +] +``` + +### Create project-level rule + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can create project approval rules using the following endpoint: + +``` +POST /projects/:id/approval_rules +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `name` | string | yes | The name of the approval rule | +| `approvals_required` | integer | yes | The number of required approvals for this rule | +| `users` | integer | no | The ids of users as approvers | +| `groups` | integer | no | The ids of groups as approvers | + +```json +{ + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" } ], - "approvals_before_merge": 2, - "reset_approvals_on_push": true, - "disable_overriding_approvers_per_merge_request": false, - "merge_requests_author_approval": false + "approvals_required": 1, + "users": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + ], + "contains_hidden_groups": false +} +``` + +### Update project-level rule + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can update project approval rules using the following endpoint: + +``` +PUT /projects/:id/approval_rules/:approval_rule_id +``` + +**Important:** Approvers and groups not in the `users`/`groups` param will be **removed** + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `approval_rule_id` | integer | yes | The ID of a approval rule | +| `name` | string | yes | The name of the approval rule | +| `approvals_required` | integer | yes | The number of required approvals for this rule | +| `users` | integer | no | The ids of users as approvers | +| `groups` | integer | no | The ids of groups as approvers | + +```json +{ + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" + } + ], + "approvals_required": 1, + "users": [ + { + "id": 2, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + ], + "contains_hidden_groups": false } ``` +### Delete project-level rule + +>**Note:** This API endpoint is only available on 12.3 Starter and above. + +You can delete project approval rules using the following endpoint: + +``` +DELETE /projects/:id/approval_rules/:approval_rule_id +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `approval_rule_id` | integer | yes | The ID of a approval rule + ### Change allowed approvers +>**Note:** This API endpoint has been deprecated. Please use Approval Rule API instead. >**Note:** This API endpoint is only available on 10.6 Starter and above. If you are allowed to, you can change approvers and approver groups using @@ -225,8 +414,6 @@ GET /projects/:id/merge_requests/:merge_request_iid/approvals } } ], - "approvers": [], - "approver_groups": [] } ``` @@ -247,7 +434,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/approvals |----------------------|---------|----------|--------------------------------------------| | `id` | integer | yes | The ID of a project | | `merge_request_iid` | integer | yes | The IID of MR | -| `approvals_required` | integer | yes | Approvals required before MR can be merged | +| `approvals_required` | integer | yes | Approvals required before MR can be merged. Deprecated in 12.0 in favor of Approval Rules API. | ```json { @@ -262,14 +449,13 @@ POST /projects/:id/merge_requests/:merge_request_iid/approvals "merge_status": "cannot_be_merged", "approvals_required": 2, "approvals_left": 2, - "approved_by": [], - "approvers": [], - "approver_groups": [] + "approved_by": [] } ``` ### Change allowed approvers for Merge Request +>**Note:** This API endpoint has been deprecated. Please use Approval Rule API instead. >**Note:** This API endpoint is only available on 10.6 Starter and above. If you are allowed to, you can change approvers and approver groups using @@ -399,8 +585,6 @@ does not match, the response code will be `409`. } } ], - "approvers": [], - "approver_groups": [] } ``` diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index de87e4a0aee..49ed4968b0d 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -44,7 +44,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead. | | `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me` | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | -| `approver_ids` **(STARTER)** | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `approver_ids` **(STARTER)** | integer array | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -206,7 +206,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13060] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced][ce-13060] in GitLab 9.5)_ | -| `approver_ids` **(STARTER)** | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `approver_ids` **(STARTER)** | integer array | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -358,7 +358,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced][ce-13060] in GitLab 9.5)_ | -| `approver_ids` **(STARTER)** | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `approver_ids` **(STARTER)** | integer array | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -1127,7 +1127,7 @@ the `approvals_before_merge` parameter: ## Delete a merge request -Only for admins and project owners. Soft deletes the merge request in question. +Only for admins and project owners. Deletes the merge request in question. ``` DELETE /projects/:id/merge_requests/:merge_request_iid @@ -1331,9 +1331,11 @@ If you don't have permissions to accept this merge request - you'll get a `401` If the merge request is already merged or closed - you get `405` and error message 'Method Not Allowed' In case the merge request is not set to be merged when the pipeline succeeds, you'll also get a `406` error. + ``` PUT /projects/:id/merge_requests/:merge_request_iid/cancel_merge_when_pipeline_succeeds ``` + Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 76e3a0fa1a4..f9382361187 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -50,11 +50,14 @@ The web application flow is: `/oauth/authorize` endpoint with the following GET parameters: ``` - https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=code&state=YOUR_UNIQUE_STATE_HASH + https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=code&state=YOUR_UNIQUE_STATE_HASH&scope=REQUESTED_SCOPES ``` - This will ask the user to approve the applications access to their account and - then redirect back to the `REDIRECT_URI` you provided. The redirect will + This will ask the user to approve the applications access to their account + based on the scopes specified in `REQUESTED_SCOPES` and then redirect back to + the `REDIRECT_URI` you provided. The [scope parameter](https://github.com/doorkeeper-gem/doorkeeper/wiki/Using-Scopes#requesting-particular-scopes) + is a space separated list of scopes you want to have access to (e.g. `scope=read_user+profile` + would request `read_user` and `profile` scopes). The redirect will include the GET `code` parameter, for example: ``` @@ -110,11 +113,14 @@ To request the access token, you should redirect the user to the `/oauth/authorize` endpoint using `token` response type: ``` -https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=token&state=YOUR_UNIQUE_STATE_HASH +https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=token&state=YOUR_UNIQUE_STATE_HASH&scope=REQUESTED_SCOPES ``` -This will ask the user to approve the application's access to their account and -then redirect them back to the `REDIRECT_URI` you provided. The redirect +This will ask the user to approve the applications access to their account +based on the scopes specified in `REQUESTED_SCOPES` and then redirect back to +the `REDIRECT_URI` you provided. The [scope parameter](https://github.com/doorkeeper-gem/doorkeeper/wiki/Using-Scopes#requesting-particular-scopes) + is a space separated list of scopes you want to have access to (e.g. `scope=read_user+profile` +would request `read_user` and `profile` scopes). The redirect will include a fragment with `access_token` as well as token details in GET parameters, for example: diff --git a/doc/api/pipeline_triggers.md b/doc/api/pipeline_triggers.md index 736312df116..e207ff8e98a 100644 --- a/doc/api/pipeline_triggers.md +++ b/doc/api/pipeline_triggers.md @@ -120,35 +120,6 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form descript } ``` -## Take ownership of a project trigger - -Update an owner of a project trigger. - -``` -POST /projects/:id/triggers/:trigger_id/take_ownership -``` - -| Attribute | Type | required | Description | -|---------------|---------|----------|--------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `trigger_id` | integer | yes | The trigger id | - -``` -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/triggers/10/take_ownership" -``` - -```json -{ - "id": 10, - "description": "my trigger", - "created_at": "2016-01-07T09:53:58.235Z", - "last_used": null, - "token": "6d056f63e50fe6f8c5f8f4aa10edb7", - "updated_at": "2016-01-07T09:53:58.235Z", - "owner": null -} -``` - ## Remove a project trigger Remove a project's build trigger. diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 614ea41d572..762a4ad95ab 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -261,7 +261,7 @@ Parameters: NOTE: **Note:** `name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added -through the ["Add an existing Kubernetes Cluster"](../user/project/clusters/index.md#adding-an-existing-kubernetes-cluster) option or +through the ["Add existing Kubernetes cluster"](../user/project/clusters/index.md#add-existing-kubernetes-cluster) option or through the ["Add existing cluster to project"](#add-existing-cluster-to-project) endpoint. Example request: diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index eab905bbc5f..591911bb8ec 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -74,7 +74,7 @@ POST /projects/:id/variables | `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | | `protected` | boolean | no | Whether the variable is protected | | `masked` | boolean | no | Whether the variable is masked | -| `environment_scope` | string | no | The `environment_scope` of the variable **(PREMIUM)** | +| `environment_scope` | string | no | The `environment_scope` of the variable | ``` curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" @@ -108,7 +108,7 @@ PUT /projects/:id/variables/:key | `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | | `protected` | boolean | no | Whether the variable is protected | | `masked` | boolean | no | Whether the variable is masked | -| `environment_scope` | string | no | The `environment_scope` of the variable **(PREMIUM)** | +| `environment_scope` | string | no | The `environment_scope` of the variable | ``` curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/NEW_VARIABLE" --form "value=updated value" diff --git a/doc/api/projects.md b/doc/api/projects.md index 781192fb92e..cf28ea84704 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -465,6 +465,194 @@ GET /users/:user_id/projects ] ``` +## List projects starred by a user + +Get a list of visible projects owned by the given user. When accessed without authentication, only public projects are returned. + +``` +GET /users/:user_id/starred_projects +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `user_id` | string | yes | The ID or username of the user. | +| `archived` | boolean | no | Limit by archived status. | +| `visibility` | string | no | Limit by visibility `public`, `internal`, or `private`. | +| `order_by` | string | no | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | +| `sort` | string | no | Return projects sorted in `asc` or `desc` order. Default is `desc`. | +| `search` | string | no | Return list of projects matching the search criteria. | +| `simple` | boolean | no | Return only limited fields for each project. This is a no-op without authentication as then _only_ simple fields are returned.. | +| `owned` | boolean | no | Limit by projects explicitly owned by the current user. | +| `membership` | boolean | no | Limit by projects that the current user is a member of. | +| `starred` | boolean | no | Limit by projects starred by the current user. | +| `statistics` | boolean | no | Include project statistics. | +| `with_custom_attributes` | boolean | no | Include [custom attributes](custom_attributes.md) in response (admins only). | +| `with_issues_enabled` | boolean | no | Limit by enabled issues feature. | +| `with_merge_requests_enabled` | boolean | no | Limit by enabled merge requests feature. | +| `min_access_level` | integer | no | Limit by current user minimal [access level](members.md). | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/5/starred_projects" +``` + +Example response: + +```json +[ + { + "id": 4, + "description": null, + "default_branch": "master", + "visibility": "private", + "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", + "web_url": "http://example.com/diaspora/diaspora-client", + "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", + "tag_list": [ + "example", + "disapora client" + ], + "owner": { + "id": 3, + "name": "Diaspora", + "created_at": "2013-09-30T13:46:02Z" + }, + "name": "Diaspora Client", + "name_with_namespace": "Diaspora / Diaspora Client", + "path": "diaspora-client", + "path_with_namespace": "diaspora/diaspora-client", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "jobs_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "resolve_outdated_diff_discussions": false, + "container_registry_enabled": false, + "created_at": "2013-09-30T13:46:02Z", + "last_activity_at": "2013-09-30T13:46:02Z", + "creator_id": 3, + "namespace": { + "id": 3, + "name": "Diaspora", + "path": "diaspora", + "kind": "group", + "full_path": "diaspora" + }, + "import_status": "none", + "archived": false, + "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0, + "runners_token": "b8547b1dc37721d05889db52fa2f02", + "public_jobs": true, + "shared_with_groups": [], + "only_allow_merge_if_pipeline_succeeds": false, + "only_allow_merge_if_all_discussions_are_resolved": false, + "request_access_enabled": false, + "merge_method": "merge", + "statistics": { + "commit_count": 37, + "storage_size": 1038090, + "repository_size": 1038090, + "lfs_objects_size": 0, + "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } + }, + { + "id": 6, + "description": null, + "default_branch": "master", + "visibility": "private", + "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", + "http_url_to_repo": "http://example.com/brightbox/puppet.git", + "web_url": "http://example.com/brightbox/puppet", + "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", + "tag_list": [ + "example", + "puppet" + ], + "owner": { + "id": 4, + "name": "Brightbox", + "created_at": "2013-09-30T13:46:02Z" + }, + "name": "Puppet", + "name_with_namespace": "Brightbox / Puppet", + "path": "puppet", + "path_with_namespace": "brightbox/puppet", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "jobs_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "resolve_outdated_diff_discussions": false, + "container_registry_enabled": false, + "created_at": "2013-09-30T13:46:02Z", + "last_activity_at": "2013-09-30T13:46:02Z", + "creator_id": 3, + "namespace": { + "id": 4, + "name": "Brightbox", + "path": "brightbox", + "kind": "group", + "full_path": "brightbox" + }, + "import_status": "none", + "import_error": null, + "permissions": { + "project_access": { + "access_level": 10, + "notification_level": 3 + }, + "group_access": { + "access_level": 50, + "notification_level": 3 + } + }, + "archived": false, + "avatar_url": null, + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0, + "runners_token": "b8547b1dc37721d05889db52fa2f02", + "public_jobs": true, + "shared_with_groups": [], + "only_allow_merge_if_pipeline_succeeds": false, + "only_allow_merge_if_all_discussions_are_resolved": false, + "request_access_enabled": false, + "merge_method": "merge", + "statistics": { + "commit_count": 12, + "storage_size": 2066080, + "repository_size": 2066080, + "lfs_objects_size": 0, + "job_artifacts_size": 0 + }, + "_links": { + "self": "http://example.com/api/v4/projects", + "issues": "http://example.com/api/v4/projects/1/issues", + "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", + "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", + "labels": "http://example.com/api/v4/projects/1/labels", + "events": "http://example.com/api/v4/projects/1/events", + "members": "http://example.com/api/v4/projects/1/members" + } + } +] +``` + ## Get single project Get a specific project. This endpoint can be accessed without authentication if @@ -664,9 +852,10 @@ Get the users list of a project. GET /projects/:id/users ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `search` | string | no | Search for specific users | +| Attribute | Type | Required | Description | +| ------------ | ------------- | -------- | ----------- | +| `search` | string | no | Search for specific users | +| `skip_users` | integer array | no | Filter out users with the specified IDs | ```json [ @@ -799,7 +988,7 @@ POST /projects/user/:user_id | `auto_devops_deploy_strategy` | string | no | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`) | | `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | | `approvals_before_merge` | integer | no | **(STARTER)** How many approvers should approve merge requests by default | -| `external_authorization_classification_label` | string | no | **(CORE ONLY)** The classification label for the project | +| `external_authorization_classification_label` | string | no | **(PREMIUM)** The classification label for the project | | `mirror` | boolean | no | **(STARTER)** Enables pull mirroring in a project | | `mirror_trigger_builds` | boolean | no | **(STARTER)** Pull mirroring triggers builds | @@ -856,7 +1045,7 @@ PUT /projects/:id | `auto_devops_deploy_strategy` | string | no | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`) | | `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | | `approvals_before_merge` | integer | no | **(STARTER)** How many approvers should approve merge request by default | -| `external_authorization_classification_label` | string | no | **(CORE ONLY)** The classification label for the project | +| `external_authorization_classification_label` | string | no | **(PREMIUM)** The classification label for the project | | `mirror` | boolean | no | **(STARTER)** Enables pull mirroring in a project | | `mirror_user_id` | integer | no | **(STARTER)** User responsible for all the activity surrounding a pull mirror event | | `mirror_trigger_builds` | boolean | no | **(STARTER)** Pull mirroring triggers builds | @@ -1155,6 +1344,51 @@ Example response: } ``` +## List Starrers of a project + +List the users who starred the specified project. + +``` +GET /projects/:id/starrers +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `search` | string | no | Search for specific users. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/starrers" +``` + +Example responses: + +```json +[ + { + "starred_since": "2019-01-28T14:47:30.642Z", + "user": + { + "id": 1, + "username": "jane_smith", + "name": "Jane Smith", + "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", + "web_url": "http://localhost:3000/jane_smith" + } + }, + "starred_since": "2018-01-02T11:40:26.570Z", + "user": + { + "id": 2, + "username": "janine_smith", + "name": "Janine Smith", + "state": "blocked", + "avatar_url": "http://gravatar.com/../e32131cd8.jpeg", + "web_url": "http://localhost:3000/janine_smith" + } +] +``` + ## Languages Get languages used in a project with percentage value. @@ -1676,18 +1910,20 @@ GET /projects/:id/push_rule "author_email_regex": "", "file_name_regex": "", "max_file_size": 5, - "commit_committer_check": false + "commit_committer_check": false, + "reject_unsigned_commits": false } ``` Users on GitLab [Premium, Silver, or higher](https://about.gitlab.com/pricing/) will also see -the `commit_committer_check` parameter: +the `commit_committer_check` and `reject_unsigned_commits` parameters: ```json { "id": 1, "project_id": 3, - "commit_committer_check": false + "commit_committer_check": false, + "reject_unsigned_commits": false ... } ``` @@ -1713,6 +1949,7 @@ POST /projects/:id/push_rule | `file_name_regex` **(STARTER)** | string | no | All commited filenames must **not** match this, e.g. `(jar|exe)$` | | `max_file_size` **(STARTER)** | integer | no | Maximum file size (MB) | | `commit_committer_check` **(PREMIUM)** | boolean | no | Users can only push commits to this repository that were committed with one of their own verified emails. | +| `reject_unsigned_commits` **(PREMIUM)** | boolean | no | Reject commit when it is not signed through GPG. | ### Edit project push rule @@ -1735,6 +1972,7 @@ PUT /projects/:id/push_rule | `file_name_regex` **(STARTER)** | string | no | All commited filenames must **not** match this, e.g. `(jar|exe)$` | | `max_file_size` **(STARTER)** | integer | no | Maximum file size (MB) | | `commit_committer_check` **(PREMIUM)** | boolean | no | Users can only push commits to this repository that were committed with one of their own verified emails. | +| `reject_unsigned_commits` **(PREMIUM)** | boolean | no | Reject commits when they are not GPG signed. | ### Delete project push rule @@ -1799,13 +2037,13 @@ Read more in the [Project Badges](project_badges.md) documentation. The non-default [issue and merge request description templates](../user/project/description_templates.md) are managed inside the project's repository. So you can manage them via the API through the [Repositories API](repositories.md) and the [Repository Files API](repository_files.md). -## Download snapshot of a git repository +## Download snapshot of a Git repository > Introduced in GitLab 10.7 This endpoint may only be accessed by an administrative user. -Download a snapshot of the project (or wiki, if requested) git repository. This +Download a snapshot of the project (or wiki, if requested) Git repository. This snapshot is always in uncompressed [tar](https://en.wikipedia.org/wiki/Tar_(computing)) format. diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index 9309306ba05..ffb4a70168b 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -185,6 +185,7 @@ Example response: { "access_level": 30, "access_level_description": "Developers + Maintainers" + } ], "unprotect_access_levels": [ { @@ -217,6 +218,7 @@ Example response: "user_id": null, "group_id": null, "access_level_description": "Developers + Maintainers" + } ], "unprotect_access_levels": [ { @@ -232,7 +234,7 @@ Example response: ### Example with user / group level access **(STARTER)** Elements in the `allowed_to_push` / `allowed_to_merge` / `allowed_to_unprotect` array should take the -form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). These access levels allow [more granular control over protected branch access](../user/project/protected_branches.md#restricting-push-and-merge-access-to-certain-users-starter) and were [added to the API in ][ee-3516] in GitLab 10.3 EE. +form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). These access levels allow [more granular control over protected branch access](../user/project/protected_branches.md#restricting-push-and-merge-access-to-certain-users-starter) and were [added to the API in](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3516) in GitLab 10.3 EE. ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&allowed_to_push%5B%5D%5Buser_id%5D=1' @@ -242,29 +244,29 @@ Example response: ```json { - "name":"*-stable", + "name": "*-stable", "push_access_levels": [ { - "access_level":null, - "user_id":1, - "group_id":null, - "access_level_description":"Administrator" + "access_level": null, + "user_id": 1, + "group_id": null, + "access_level_description": "Administrator" } ], "merge_access_levels": [ { - "access_level":40, - "user_id":null, - "group_id":null, - "access_level_description":"Maintainers" + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" } ], "unprotect_access_levels": [ { - "access_level":40, - "user_id":null, - "group_id":null, - "access_level_description":"Maintainers" + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" } ] } @@ -286,5 +288,3 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" 'https://git | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the branch | - -[ee-3516]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3516 "ProtectedBranches API handles per user/group granularity" diff --git a/doc/api/protected_tags.md b/doc/api/protected_tags.md index 3adca61a108..fb6fa040244 100644 --- a/doc/api/protected_tags.md +++ b/doc/api/protected_tags.md @@ -5,6 +5,7 @@ **Valid access levels** Currently, these levels are recognized: + ``` 0 => No access 30 => Developer access diff --git a/doc/api/releases/img/upcoming_release_v12_1.png b/doc/api/releases/img/upcoming_release_v12_1.png Binary files differnew file mode 100644 index 00000000000..cc3070fd19d --- /dev/null +++ b/doc/api/releases/img/upcoming_release_v12_1.png diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index e7f79a0d359..e74b35fd959 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -6,7 +6,7 @@ ## List Releases -Paginated list of Releases, sorted by `created_at`. +Paginated list of Releases, sorted by `released_at`. ``` GET /projects/:id/releases @@ -32,6 +32,7 @@ Example response: "name":"Awesome app v0.2 beta", "description_html":"\u003ch2 dir=\"auto\"\u003e\n\u003ca id=\"user-content-changelog\" class=\"anchor\" href=\"#changelog\" aria-hidden=\"true\"\u003e\u003c/a\u003eCHANGELOG\u003c/h2\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eEscape label and milestone titles to prevent XSS in GFM autocomplete. !2740\u003c/li\u003e\n\u003cli\u003ePrevent private snippets from being embeddable.\u003c/li\u003e\n\u003cli\u003eAdd subresources removal to member destroy service.\u003c/li\u003e\n\u003c/ul\u003e", "created_at":"2019-01-03T01:56:19.539Z", + "released_at":"2019-01-03T01:56:19.539Z", "author":{ "id":1, "name":"Administrator", @@ -98,6 +99,7 @@ Example response: "name":"Awesome app v0.1 alpha", "description_html":"\u003ch2 dir=\"auto\"\u003e\n\u003ca id=\"user-content-changelog\" class=\"anchor\" href=\"#changelog\" aria-hidden=\"true\"\u003e\u003c/a\u003eCHANGELOG\u003c/h2\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eRemove limit of 100 when searching repository code. !8671\u003c/li\u003e\n\u003cli\u003eShow error message when attempting to reopen an MR and there is an open MR for the same branch. !16447 (Akos Gyimesi)\u003c/li\u003e\n\u003cli\u003eFix a bug where internal email pattern wasn't respected. !22516\u003c/li\u003e\n\u003c/ul\u003e", "created_at":"2019-01-03T01:55:18.203Z", + "released_at":"2019-01-03T01:55:18.203Z", "author":{ "id":1, "name":"Administrator", @@ -178,6 +180,7 @@ Example response: "name":"Awesome app v0.1 alpha", "description_html":"\u003ch2 dir=\"auto\"\u003e\n\u003ca id=\"user-content-changelog\" class=\"anchor\" href=\"#changelog\" aria-hidden=\"true\"\u003e\u003c/a\u003eCHANGELOG\u003c/h2\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eRemove limit of 100 when searching repository code. !8671\u003c/li\u003e\n\u003cli\u003eShow error message when attempting to reopen an MR and there is an open MR for the same branch. !16447 (Akos Gyimesi)\u003c/li\u003e\n\u003cli\u003eFix a bug where internal email pattern wasn't respected. !22516\u003c/li\u003e\n\u003c/ul\u003e", "created_at":"2019-01-03T01:55:18.203Z", + "released_at":"2019-01-03T01:55:18.203Z", "author":{ "id":1, "name":"Administrator", @@ -247,6 +250,7 @@ POST /projects/:id/releases | `assets:links`| array of hash | no | An array of assets links. | | `assets:links:name`| string | no (if `assets:links` specified, it's required) | The name of the link. | | `assets:links:url`| string | no (if `assets:links` specified, it's required) | The url of the link. | +| `released_at` | datetime | no | The date when the release will be/was ready. Defaults to the current time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | Example request: @@ -265,6 +269,7 @@ Example response: "name":"New release", "description_html":"\u003cp dir=\"auto\"\u003eSuper nice release\u003c/p\u003e", "created_at":"2019-01-03T02:22:45.118Z", + "released_at":"2019-01-03T02:22:45.118Z", "author":{ "id":1, "name":"Administrator", @@ -335,6 +340,7 @@ PUT /projects/:id/releases/:tag_name | `tag_name` | string | yes | The tag where the release will be created from. | | `name` | string | no | The release name. | | `description` | string | no | The description of the release. You can use [markdown](../../user/markdown.md). | +| `released_at` | datetime | no | The date when the release will be/was ready. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | Example request: @@ -351,6 +357,7 @@ Example response: "name":"new name", "description_html":"\u003ch2 dir=\"auto\"\u003e\n\u003ca id=\"user-content-changelog\" class=\"anchor\" href=\"#changelog\" aria-hidden=\"true\"\u003e\u003c/a\u003eCHANGELOG\u003c/h2\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eRemove limit of 100 when searching repository code. !8671\u003c/li\u003e\n\u003cli\u003eShow error message when attempting to reopen an MR and there is an open MR for the same branch. !16447 (Akos Gyimesi)\u003c/li\u003e\n\u003cli\u003eFix a bug where internal email pattern wasn't respected. !22516\u003c/li\u003e\n\u003c/ul\u003e", "created_at":"2019-01-03T01:55:18.203Z", + "released_at":"2019-01-03T01:55:18.203Z", "author":{ "id":1, "name":"Administrator", @@ -430,6 +437,7 @@ Example response: "name":"new name", "description_html":"\u003ch2 dir=\"auto\"\u003e\n\u003ca id=\"user-content-changelog\" class=\"anchor\" href=\"#changelog\" aria-hidden=\"true\"\u003e\u003c/a\u003eCHANGELOG\u003c/h2\u003e\n\u003cul dir=\"auto\"\u003e\n\u003cli\u003eRemove limit of 100 when searching repository code. !8671\u003c/li\u003e\n\u003cli\u003eShow error message when attempting to reopen an MR and there is an open MR for the same branch. !16447 (Akos Gyimesi)\u003c/li\u003e\n\u003cli\u003eFix a bug where internal email pattern wasn't respected. !22516\u003c/li\u003e\n\u003c/ul\u003e", "created_at":"2019-01-03T01:55:18.203Z", + "released_at":"2019-01-03T01:55:18.203Z", "author":{ "id":1, "name":"Administrator", @@ -480,3 +488,11 @@ Example response: } } ``` + +## Upcoming Releases + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/38105) in GitLab 12.1. + +A release with a `released_at` attribute set to a future date will be labeled an **Upcoming Release** in the UI: + + diff --git a/doc/api/releases/links.md b/doc/api/releases/links.md index 9c91264ed65..b1b75abe32f 100644 --- a/doc/api/releases/links.md +++ b/doc/api/releases/links.md @@ -21,7 +21,7 @@ GET /projects/:id/releases/:tag_name/assets/links Example request: ```sh -curl --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "http://localhost:3000/api/v4/projects/24/releases/v0.1/assets/links" +curl --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links" ``` Example response: @@ -60,7 +60,7 @@ GET /projects/:id/releases/:tag_name/assets/links/:link_id Example request: ```sh -curl --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "http://localhost:3000/api/v4/projects/24/releases/v0.1/assets/links/1" +curl --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links/1" ``` Example response: @@ -96,7 +96,7 @@ curl --request POST \ --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" \ --data name="awesome-v0.2.dmg" \ --data url="http://192.168.10.15:3000" \ - "http://localhost:3000/api/v4/projects/24/releases/v0.1/assets/links" + "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links" ``` Example response: @@ -132,7 +132,7 @@ You have to specify at least one of `name` or `url` Example request: ```sh -curl --request PUT --data name="new name" --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "http://localhost:3000/api/v4/projects/24/releases/v0.1/assets/links/1" +curl --request PUT --data name="new name" --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links/1" ``` Example response: @@ -163,7 +163,7 @@ DELETE /projects/:id/releases/:tag_name/assets/links/:link_id Example request: ```sh -curl --request DELETE --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "http://localhost:3000/api/v4/projects/24/releases/v0.1/assets/links/1" +curl --request DELETE --header "PRIVATE-TOKEN: gDybLx3yrUK_HLp3qPjS" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links/1" ``` Example response: diff --git a/doc/api/repositories.md b/doc/api/repositories.md index ffae5c17310..4aff79c9c62 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -235,17 +235,17 @@ Example response: ```json { - "id": "1a0b36b3cdad1d2ee32457c102a8c0b7056fa863", - "short_id": "1a0b36b3", - "title": "Initial commit", - "created_at": "2014-02-27T08:03:18.000Z", - "parent_ids": [], - "message": "Initial commit\n", - "author_name": "Dmitriy Zaporozhets", - "author_email": "dmitriy.zaporozhets@gmail.com", - "authored_date": "2014-02-27T08:03:18.000Z", - "committer_name": "Dmitriy Zaporozhets", - "committer_email": "dmitriy.zaporozhets@gmail.com", - "committed_date": "2014-02-27T08:03:18.000Z" + "id": "1a0b36b3cdad1d2ee32457c102a8c0b7056fa863", + "short_id": "1a0b36b3", + "title": "Initial commit", + "created_at": "2014-02-27T08:03:18.000Z", + "parent_ids": [], + "message": "Initial commit\n", + "author_name": "Dmitriy Zaporozhets", + "author_email": "dmitriy.zaporozhets@gmail.com", + "authored_date": "2014-02-27T08:03:18.000Z", + "committer_name": "Dmitriy Zaporozhets", + "committer_email": "dmitriy.zaporozhets@gmail.com", + "committed_date": "2014-02-27T08:03:18.000Z" } ``` diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 87c7f371de1..513dc996c91 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -80,6 +80,75 @@ X-Gitlab-Size: 1476 ... ``` +## Get file blame from repository + +Allows you to receive blame information. Each blame range contains lines and corresponding commit info. + +``` +GET /projects/:id/repository/files/:file_path/blame +``` + +```bash +curl --request GET --header 'PRIVATE-TOKEN: <your_access_token>' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=master' +``` + +Example response: + +```json +[ + { + "commit": { + "id": "d42409d56517157c48bf3bd97d3f75974dde19fb", + "message": "Add feature\n\nalso fix bug\n", + "parent_ids": [ + "cc6e14f9328fa6d7b5a0d3c30dc2002a3f2a3822" + ], + "authored_date": "2015-12-18T08:12:22.000Z", + "author_name": "John Doe", + "author_email": "john.doe@example.com", + "committed_date": "2015-12-18T08:12:22.000Z", + "committer_name": "John Doe", + "committer_email": "john.doe@example.com" + }, + "lines": [ + "require 'fileutils'", + "require 'open3'", + "" + ] + }, + ... +] +``` + +Parameters: + +- `file_path` (required) - Url encoded full path to new file. Ex. lib%2Fclass%2Erb +- `ref` (required) - The name of branch, tag or commit + +NOTE: **Note:** +`HEAD` method return just file metadata as in [Get file from repository](repository_files.md#get-file-from-repository). + +```bash +curl --head --header 'PRIVATE-TOKEN: <your_access_token>' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/path%2Fto%2Ffile.rb/blame?ref=master' +``` + +Example response: + +```text +HTTP/1.1 200 OK +... +X-Gitlab-Blob-Id: 79f7bbd25901e8334750839545a9bd021f0e4c83 +X-Gitlab-Commit-Id: d5a3ff139356ce33e37e73add446f16869741b50 +X-Gitlab-Content-Sha256: 4c294617b60715c1d218e61164a3abd4808a4284cbc30e6728a01ad9aada4481 +X-Gitlab-Encoding: base64 +X-Gitlab-File-Name: file.rb +X-Gitlab-File-Path: path/to/file.rb +X-Gitlab-Last-Commit-Id: 570e7b2abdd848b95f2f578043fc23bd6f6fd24d +X-Gitlab-Ref: master +X-Gitlab-Size: 1476 +... +``` + ## Get raw file from repository ``` @@ -177,7 +246,7 @@ error message. Possible causes for a failed commit include: user tried to make an empty commit; - the branch was updated by a Git push while the file edit was in progress. -Currently gitlab-shell has a boolean return code, preventing GitLab from specifying the error. +Currently GitLab Shell has a boolean return code, preventing GitLab from specifying the error. ## Delete existing file in repository diff --git a/doc/api/repository_submodules.md b/doc/api/repository_submodules.md index 2c44c4abc93..5a722a75cb9 100644 --- a/doc/api/repository_submodules.md +++ b/doc/api/repository_submodules.md @@ -43,7 +43,7 @@ Example response: "ae1d9fb46aa2b07ee9836d49862ec4e2c46fbbba" ], "committed_date": "2018-09-20T09:26:24.000-07:00", - "authored_date": "2018-09-20T09:26:24.000-07:00", + "authored_date": "2018-09-20T09:26:24.000-07:00", "status": null } ``` diff --git a/doc/api/services.md b/doc/api/services.md index df15e6892b0..7d025cd3bdf 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -595,44 +595,6 @@ Remove all previously Jira settings from a project. DELETE /projects/:id/services/jira ``` -## Kubernetes - -Kubernetes / OpenShift integration - -CAUTION: **Warning:** -Kubernetes service integration has been deprecated in GitLab 10.3. API service endpoints will continue to work as long as the Kubernetes service is active, however if the service is inactive API endpoints will automatically return a `400 Bad Request`. Read [GitLab 10.3 release post](https://about.gitlab.com/2017/12/22/gitlab-10-3-released/#kubernetes-integration-service) for more information. - -### Create/Edit Kubernetes service - -Set Kubernetes service for a project. - -``` -PUT /projects/:id/services/kubernetes -``` - -Parameters: - -- `namespace` (**required**) - The Kubernetes namespace to use -- `api_url` (**required**) - The URL to the Kubernetes cluster API. For example, `https://kubernetes.example.com` -- `token` (**required**) - The service token to authenticate against the Kubernetes cluster with -- `ca_pem` (optional) - A custom certificate authority bundle to verify the Kubernetes cluster with (PEM format) - -### Delete Kubernetes service - -Delete Kubernetes service for a project. - -``` -DELETE /projects/:id/services/kubernetes -``` - -### Get Kubernetes service settings - -Get Kubernetes service settings for a project. - -``` -GET /projects/:id/services/kubernetes -``` - ## Slack slash commands Ability to receive slash commands from a Slack chat instance. @@ -972,22 +934,28 @@ Parameters: | `channel` | string | false | Default channel to use if others are not configured | | `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | | `notify_only_default_branch` | boolean | false | Send notifications only for the default branch | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | +| `commit_events` | boolean | false | Enable notifications for commit events | +| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | | `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `confidential_note_channel` | string | false | The name of the channel to receive confidential note events notifications | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `deployment_channel` | string | false | The name of the channel to receive deployment events notifications | +| `deployment_events` | boolean | false | Enable notifications for deployment events | +| `issue_channel` | string | false | The name of the channel to receive issues events notifications | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `job_events` | boolean | false | Enable notifications for job events | +| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | | `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_channel` | string | false | The name of the channel to receive note events notifications | | `note_events` | boolean | false | Enable notifications for note events | +| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | | `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | | `push_channel` | string | false | The name of the channel to receive push events notifications | -| `issue_channel` | string | false | The name of the channel to receive issues events notifications | -| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | -| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | -| `note_channel` | string | false | The name of the channel to receive note events notifications | +| `push_events` | boolean | false | Enable notifications for push events | | `tag_push_channel` | string | false | The name of the channel to receive tag push events notifications | -| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | | `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | ### Delete Slack service diff --git a/doc/api/settings.md b/doc/api/settings.md index ff48cac1f47..710b63c9a2f 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -39,6 +39,7 @@ Example response: "session_expire_delay" : 10080, "home_page_url" : null, "default_snippet_visibility" : "private", + "outbound_local_requests_whitelist": [], "domain_whitelist" : [], "domain_blacklist_enabled" : false, "domain_blacklist" : [], @@ -63,7 +64,10 @@ Example response: "performance_bar_allowed_group_id": 42, "instance_statistics_visibility_private": false, "user_show_add_ssh_key_message": true, - "local_markdown_version": 0 + "local_markdown_version": 0, + "allow_local_requests_from_hooks_and_services": true, + "allow_local_requests_from_web_hooks_and_services": true, + "allow_local_requests_from_system_hooks": false } ``` @@ -113,6 +117,7 @@ Example response: "default_project_visibility": "internal", "default_snippet_visibility": "private", "default_group_visibility": "private", + "outbound_local_requests_whitelist": [], "domain_whitelist": [], "domain_blacklist_enabled" : false, "domain_blacklist" : [], @@ -136,7 +141,10 @@ Example response: "user_show_add_ssh_key_message": true, "file_template_project_id": 1, "local_markdown_version": 0, - "geo_node_allowed_ips": "0.0.0.0/0, ::/0" + "geo_node_allowed_ips": "0.0.0.0/0, ::/0", + "allow_local_requests_from_hooks_and_services": true, + "allow_local_requests_from_web_hooks_and_services": true, + "allow_local_requests_from_system_hooks": false } ``` @@ -175,7 +183,9 @@ are listed in the descriptions of the relevant settings. | `akismet_api_key` | string | required by: `akismet_enabled` | API key for akismet spam protection. | | `akismet_enabled` | boolean | no | (**If enabled, requires:** `akismet_api_key`) Enable or disable akismet spam protection. | | `allow_group_owners_to_manage_ldap` | boolean | no | **(PREMIUM)** Set to `true` to allow group owners to manage LDAP | -| `allow_local_requests_from_hooks_and_services` | boolean | no | Allow requests to the local network from hooks and services. | +| `allow_local_requests_from_hooks_and_services` | boolean | no | (Deprecated: Use `allow_local_requests_from_web_hooks_and_services` instead) Allow requests to the local network from hooks and services. | +| `allow_local_requests_from_web_hooks_and_services` | boolean | no | Allow requests to the local network from web hooks and services. | +| `allow_local_requests_from_system_hooks` | boolean | no | Allow requests to the local network from system hooks. | | `authorized_keys_enabled` | boolean | no | By default, we write to the `authorized_keys` file to support Git over SSH without additional configuration. GitLab can be optimized to authenticate SSH keys via the database file. Only disable this if you have configured your OpenSSH server to use the AuthorizedKeysCommand. | | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It will automatically build, test, and deploy applications based on a predefined CI/CD configuration. | @@ -193,6 +203,7 @@ are listed in the descriptions of the relevant settings. | `domain_blacklist` | array of strings | required by: `domain_blacklist_enabled` | Users with e-mail addresses that match these domain(s) will NOT be able to sign-up. Wildcards allowed. Use separate lines for multiple entries. Ex: `domain.com`, `*.domain.com`. | | `domain_blacklist_enabled` | boolean | no | (**If enabled, requires:** `domain_blacklist`) Allows blocking sign-ups from emails from specific domains. | | `domain_whitelist` | array of strings | no | Force people to use only corporate emails for sign-up. Default is `null`, meaning there is no restriction. | +| `outbound_local_requests_whitelist` | array of strings | no | Define a list of trusted domains or ip addresses to which local requests are allowed when local requests for hooks and services are disabled. | `dsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded DSA key. Default is `0` (no restriction). `-1` disables DSA keys. | | `ecdsa_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ECDSA key. Default is `0` (no restriction). `-1` disables ECDSA keys. | | `ed25519_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ED25519 key. Default is `0` (no restriction). `-1` disables ED25519 keys. | @@ -221,19 +232,19 @@ are listed in the descriptions of the relevant settings. | `file_template_project_id` | integer | no | **(PREMIUM)** The ID of a project to load custom file templates from | | `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `geo_status_timeout` | integer | no | **(PREMIUM)** The amount of seconds after which a request to get a secondary node status will time out. | -| `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | +| `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for Git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | | `gravatar_enabled` | boolean | no | Enable Gravatar. | | `hashed_storage_enabled` | boolean | no | Create new projects using hashed storage paths: Enable immutable, hash-based paths and repository names to store repositories on disk. This prevents repositories from having to be moved or renamed when the Project URL changes and may improve disk I/O performance. (EXPERIMENTAL) | | `help_page_hide_commercial_content` | boolean | no | Hide marketing-related entries from help. | -| `help_page_support_url` | string | no | Alternate support URL for help page. | +| `help_page_support_url` | string | no | Alternate support URL for help page and help dropdown. | | `help_page_text` | string | no | Custom text displayed on the help page. | | `help_text` | string | no | **(PREMIUM)** GitLab server administrator information | | `hide_third_party_offers` | boolean | no | Do not display offers from third parties within GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | -| `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable git housekeeping. | +| `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable Git housekeeping. | | `housekeeping_full_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | | `housekeeping_gc_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which `git gc` is run. | | `housekeeping_incremental_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | @@ -310,4 +321,8 @@ are listed in the descriptions of the relevant settings. | `user_show_add_ssh_key_message` | boolean | no | When set to `false` disable the "You won't be able to pull or push project code via SSH" warning shown to users with no uploaded SSH key. | | `version_check_enabled` | boolean | no | Let GitLab inform you when an update is available. | | `local_markdown_version` | integer | no | Increase this value when any cached markdown should be invalidated. | +| `snowplow_enabled` | boolean | no | Enable snowplow tracking. | +| `snowplow_collector_hostname` | string | required by: `snowplow_enabled` | The Snowplow collector hostname. (e.g. `snowplow.trx.gitlab.net`) | +| `snowplow_site_id` | string | no | The Snowplow site name / application id. (e.g. `gitlab`) | +| `snowplow_cookie_domain` | string | no | The Snowplow cookie domain. (e.g. `.gitlab.com`) | | `geo_node_allowed_ips` | string | yes | **(PREMIUM)** Comma-separated list of IPs and CIDRs of allowed secondary nodes. For example, `1.1.1.1, 2.2.2.0/24`. | diff --git a/doc/api/system_hooks.md b/doc/api/system_hooks.md index f8563e819db..1b8c23b8e2d 100644 --- a/doc/api/system_hooks.md +++ b/doc/api/system_hooks.md @@ -11,8 +11,6 @@ Read more about [system hooks](../system_hooks/system_hooks.md). Get a list of all system hooks. ---- - ``` GET /hooks ``` @@ -44,8 +42,6 @@ Example response: Add a new system hook. ---- - ``` POST /hooks ``` @@ -116,8 +112,6 @@ Example response: Deletes a system hook. ---- - ``` DELETE /hooks/:id ``` @@ -130,4 +124,4 @@ Example request: ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/hooks/2 -```
\ No newline at end of file +``` diff --git a/doc/api/tags.md b/doc/api/tags.md index 3177fec618f..1d874fea1f8 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -112,7 +112,7 @@ Parameters: - `tag_name` (required) - The name of a tag - `ref` (required) - Create tag using commit SHA, another tag name, or branch name. - `message` (optional) - Creates annotated tag. -- `release_description` (optional) - Add release notes to the git tag and store it in the GitLab database. +- `release_description` (optional) - Add release notes to the Git tag and store it in the GitLab database. ```json { @@ -141,6 +141,7 @@ Parameters: "message": null } ``` + The message will be `null` when creating a lightweight tag otherwise it will contain the annotation. @@ -165,7 +166,7 @@ Parameters: ## Create a new release -Add release notes to the existing git tag. If there +Add release notes to the existing Git tag. If there already exists a release for the given tag, status code `409` is returned. ``` diff --git a/doc/api/users.md b/doc/api/users.md index 213d1865aca..b41fd106fc5 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -147,6 +147,21 @@ GET /users ] ``` +Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see the `shared_runners_minutes_limit`, `extra_shared_runners_minutes_limit`, and `note` parameters. + +```json +[ + { + "id": 1, + ... + "shared_runners_minutes_limit": 133, + "extra_shared_runners_minutes_limit": 133, + "note": "DMCA Request: 2018-11-05 | DMCA Violation | Abuse | https://gitlab.zendesk.com/agent/tickets/123" + ... + } +] +``` + Users on GitLab [Silver or higher](https://about.gitlab.com/pricing/) will also see the `group_saml` provider option: @@ -284,14 +299,15 @@ Example Responses: ``` Users on GitLab [Starter, Bronze, or higher](https://about.gitlab.com/pricing/) will also see -the `shared_runners_minutes_limit` and `extra_shared_runners_minutes_limit` parameters. +the `shared_runners_minutes_limit`, `extra_shared_runners_minutes_limit`, and `note` parameters. ```json { "id": 1, "username": "john_smith", "shared_runners_minutes_limit": 133, - "extra_shared_runners_minutes_limit": 133 + "extra_shared_runners_minutes_limit": 133, + "note": "DMCA Request: 2018-11-05 | DMCA Violation | Abuse | https://gitlab.zendesk.com/agent/tickets/123" ... } ``` @@ -304,7 +320,8 @@ see the `group_saml` option: "id": 1, "username": "john_smith", "shared_runners_minutes_limit": 133, - "extra_shared_runners_minutes_limit": 133 + "extra_shared_runners_minutes_limit": 133, + "note": "DMCA Request: 2018-11-05 | DMCA Violation | Abuse | https://gitlab.zendesk.com/agent/tickets/123" "identities": [ {"provider": "github", "extern_uid": "2435223452345"}, {"provider": "bitbucket", "extern_uid": "john.smith"}, @@ -332,6 +349,9 @@ Note that `force_random_password` and `reset_password` take priority over `password`. In addition, `reset_password` and `force_random_password` can be used together. +NOTE: **Note:** +From [GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29888/), `private_profile` will default to `false`. + ``` POST /users ``` @@ -357,9 +377,9 @@ Parameters: - `admin` (optional) - User is admin - true or false (default) - `can_create_group` (optional) - User can create groups - true or false - `skip_confirmation` (optional) - Skip confirmation - true or false (default) -- `external` (optional) - Flags the user as external - true or false(default) +- `external` (optional) - Flags the user as external - true or false (default) - `avatar` (optional) - Image file for user's avatar -- `private_profile` (optional) - User's profile is private - true or false +- `private_profile` (optional) - User's profile is private - true or false (default) - `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **(STARTER)** - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **(STARTER)** @@ -392,13 +412,14 @@ Parameters: - `admin` (optional) - User is admin - true or false (default) - `can_create_group` (optional) - User can create groups - true or false - `skip_reconfirmation` (optional) - Skip reconfirmation - true or false (default) -- `external` (optional) - Flags the user as external - true or false(default) +- `external` (optional) - Flags the user as external - true or false (default) - `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user - `avatar` (optional) - Image file for user's avatar -- `private_profile` (optional) - User's profile is private - true or false +- `private_profile` (optional) - User's profile is private - true or false (default) - `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user **(STARTER)** - `extra_shared_runners_minutes_limit` (optional) - Extra pipeline minutes quota for this user **(STARTER)** +- `note` (optional) - Admin notes for this user **(STARTER)** On password update, user will be forced to change it upon next login. Note, at the moment this method does only return a `404` error, @@ -595,7 +616,7 @@ Example responses ## User counts -Get the counts (same as in top right menu) of the currently signed in user. +Get the counts (same as in top right menu) of the currently signed in user. | Attribute | Type | Description | | --------- | ---- | ----------- | diff --git a/doc/api/vulnerabilities.md b/doc/api/vulnerabilities.md index cc6e5d3960b..eaa4c13de55 100644 --- a/doc/api/vulnerabilities.md +++ b/doc/api/vulnerabilities.md @@ -30,6 +30,7 @@ GET /projects/:id/vulnerabilities?scope=all GET /projects/:id/vulnerabilities?scope=dismissed GET /projects/:id/vulnerabilities?severity=high GET /projects/:id/vulnerabilities?confidence=unknown,experimental +GET /projects/:id/vulnerabilities?pipeline_id=42 ``` | Attribute | Type | Required | Description | @@ -39,6 +40,7 @@ GET /projects/:id/vulnerabilities?confidence=unknown,experimental | `scope` | string | no | Returns vulnerabilities for the given scope: `all` or `dismissed`. Defaults to `dismissed` | | `severity` | string array | no | Returns vulnerabilities belonging to specified severity level: `undefined`, `info`, `unknown`, `low`, `medium`, `high`, or `critical`. Defaults to all' | | `confidence` | string array | no | Returns vulnerabilities belonging to specified confidence level: `undefined`, `ignore`, `unknown`, `experimental`, `low`, `medium`, `high`, or `confirmed`. Defaults to all | +| `pipeline_id` | integer/string | no | Returns vulnerabilities belonging to specified pipeline. | ```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/vulnerabilities diff --git a/doc/ci/README.md b/doc/ci/README.md index 7048ceaac41..4be13204227 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -13,6 +13,11 @@ through the [continuous methodologies](introduction/index.md#introduction-to-cic - Continuous Delivery (CD) - Continuous Deployment (CD) +NOTE: **Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more.** +Watch our +["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to learn about continuous methods and how GitLab’s built-in CI can help you simplify and scale software development. + ## Overview Continuous Integration works by pushing small code chunks to your @@ -48,6 +53,9 @@ the following documents: - [GitLab CI/CD basic workflow](introduction/index.md#basic-cicd-workflow). - [Step-by-step guide for writing `.gitlab-ci.yml` for the first time](../user/project/pages/getting_started_part_four.md). +If you're coming over from Jenkins, you can also check out our handy [reference](jenkins/index.md) +for converting your pipelines. + You can also get started by using one of the [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates) available through the UI. You can use them by creating a new file, @@ -59,11 +67,11 @@ to your needs: For a broader overview, see the [CI/CD getting started](quick_start/README.md) guide. Once you're familiar with how GitLab CI/CD works, see the -[`. gitlab-ci.yml` full reference](yaml/README.md) +[`.gitlab-ci.yml` full reference](yaml/README.md) for all the attributes you can set and use. NOTE: **Note:** -GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/admin_area/settings/continuous_integration.md#extra-shared-runners-pipeline-minutes-quota-free-only). +GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](../user/gitlab_com/index.md#shared-runners). ## Configuration @@ -123,7 +131,7 @@ Its feature set is listed on the table below according to DevOps stages. | **Secure** || | [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities.| | [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | -| [License Management](../user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | +| [License Compliance](../user/application_security/license_management/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | | [Security Test reports](../user/project/merge_requests/index.md#security-reports-ultimate) **(ULTIMATE)** | Check for app vulnerabilities. | ## Examples @@ -155,6 +163,7 @@ for your CI/CD infrastructure: - [Why we chose GitLab CI for our CI/CD solution](https://about.gitlab.com/2016/10/17/gitlab-ci-oohlala/) - [Building our web-app on GitLab CI](https://about.gitlab.com/2016/07/22/building-our-web-app-on-gitlab-ci/) +- [5 Teams that made the switch to GitLab CI/CD](https://about.gitlab.com/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. @@ -165,7 +174,7 @@ been necessary. These are: #### 12.0 -- [Use refspec to clone/fetch git +- [Use refspec to clone/fetch Git repository](https://gitlab.com/gitlab-org/gitlab-runner/issues/4069). - [Old cache configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4070). diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 9a5a3624c73..a59a0477b80 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -8,7 +8,7 @@ GitLab CI/CD provides a caching mechanism that can be used to save time when your jobs are running. Caching is about speeding the time a job is executed by reusing the same -content of a previous job. It can be particularly useful when your are +content of a previous job. It can be particularly useful when you are developing software that depends on other libraries which are fetched via the internet during build time. @@ -172,6 +172,29 @@ job: cache: {} ``` +### Inherit global config, but override specific settings per job + +You can override cache settings without overwriting the global cache by using +[anchors](../yaml/README.md#anchors). For example, if you want to override the +`policy` for one job: + +```yaml +cache: &global_cache + key: ${CI_COMMIT_REF_SLUG} + paths: + - node_modules/ + - public/ + - vendor/ + policy: pull-push + +job: + cache: + # inherit all global cache settings + <<: *global_cache + # override the policy + policy: pull +``` + For more fine tuning, read also about the [`cache: policy`](../yaml/README.md#cachepolicy). @@ -378,8 +401,8 @@ Here's what happens behind the scenes: 1. `script` is executed. 1. `after_script` is executed. 1. `cache` runs and the `vendor/` directory is zipped into `cache.zip`. - This file is then saved in the directory based on the - [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. + This file is then saved in the directory based on the + [Runner's setting](#where-the-caches-are-stored) and the `cache: key`. 1. `job B` runs. 1. The cache is extracted (if found). 1. `before_script` is executed. @@ -520,7 +543,7 @@ via GitLab's UI: 1. Navigate to your project's **CI/CD > Pipelines** page. 1. Click on the **Clear Runner caches** button to clean up the cache. -  +  1. On the next push, your CI/CD job will use a new cache. diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index 241134783da..29d4f93f02e 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -4,14 +4,14 @@ type: index, concepts, howto # GitLab ChatOps -> **Notes:** -> -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4466) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24780) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. -> -> - ChatOps is currently in alpha, with some important features missing like access control. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4466) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24780) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. GitLab ChatOps provides a method to interact with CI/CD jobs through chat services like Slack. Many organizations' discussion, collaboration, and troubleshooting is taking place in chat services these days, and having a method to run CI/CD jobs with output posted back to the channel can significantly augment a team's workflow. +NOTE: **Note:** +ChatOps is currently in alpha with some important features missing, like access control. + ## How it works GitLab ChatOps is built upon two existing features: diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index b3110b435db..54b21939116 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -14,9 +14,9 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and create the project. -  +  - GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. + GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. 1. In GitLab create a [Personal Access Token](../../user/profile/personal_access_tokens.md) @@ -26,127 +26,127 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. In Bitbucket, from **Settings > Webhooks**, create a new web hook to notify GitLab of new commits. - The web hook URL should be set to the GitLab API to trigger pull mirroring, - using the Personal Access Token we just generated for authentication. + The web hook URL should be set to the GitLab API to trigger pull mirroring, + using the Personal Access Token we just generated for authentication. - ```text - https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> - ``` + ```text + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` - The web hook Trigger should be set to 'Repository Push'. + The web hook Trigger should be set to 'Repository Push'. -  +  - After saving, test the web hook by pushing a change to your Bitbucket - repository. + After saving, test the web hook by pushing a change to your Bitbucket + repository. 1. In Bitbucket, create an **App Password** from **Bitbucket Settings > App Passwords** to authenticate the build status script setting commit build statuses in Bitbucket. Repository write permissions are required. -  +  1. In GitLab, from **Settings > CI/CD > Environment variables**, add variables to allow communication with Bitbucket via the Bitbucket API: - `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above. + `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above. - `BITBUCKET_USERNAME`: the username of the Bitbucket account. + `BITBUCKET_USERNAME`: the username of the Bitbucket account. - `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ. + `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ. - `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ. + `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ. 1. In Bitbucket, add a script to push the pipeline status to Bitbucket. - > Note: changes made in GitLab will be overwritten by any changes made - > upstream in Bitbucket. - - Create a file `build_status` and insert the script below and run - `chmod +x build_status` in your terminal to make the script executable. - - ```bash - #!/usr/bin/env bash - - # Push GitLab CI/CD build status to Bitbucket Cloud - - if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then - echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" - exit 1 - fi - if [ -z "$BITBUCKET_USERNAME" ]; then - echo "ERROR: BITBUCKET_USERNAME is not set" - exit 1 - fi - if [ -z "$BITBUCKET_NAMESPACE" ]; then - echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" - BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE - fi - if [ -z "$BITBUCKET_REPOSITORY" ]; then - echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" - BITBUCKET_REPOSITORY=$CI_PROJECT_NAME - fi - - BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" - BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" - BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" - - case "$BUILD_STATUS" in - running) - BITBUCKET_STATE="INPROGRESS" - BITBUCKET_DESCRIPTION="The build is running!" - ;; - passed) - BITBUCKET_STATE="SUCCESSFUL" - BITBUCKET_DESCRIPTION="The build passed!" - ;; - failed) - BITBUCKET_STATE="FAILED" - BITBUCKET_DESCRIPTION="The build failed." - ;; - esac - - echo "Pushing status to $BITBUCKET_STATUS_API..." - curl --request POST $BITBUCKET_STATUS_API \ - --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ - --header "Content-Type:application/json" \ - --silent \ - --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": - \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" - ``` + > Note: changes made in GitLab will be overwritten by any changes made + > upstream in Bitbucket. + + Create a file `build_status` and insert the script below and run + `chmod +x build_status` in your terminal to make the script executable. + + ```bash + #!/usr/bin/env bash + + # Push GitLab CI/CD build status to Bitbucket Cloud + + if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then + echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" + exit 1 + fi + if [ -z "$BITBUCKET_USERNAME" ]; then + echo "ERROR: BITBUCKET_USERNAME is not set" + exit 1 + fi + if [ -z "$BITBUCKET_NAMESPACE" ]; then + echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" + BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE + fi + if [ -z "$BITBUCKET_REPOSITORY" ]; then + echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" + BITBUCKET_REPOSITORY=$CI_PROJECT_NAME + fi + + BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" + BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" + BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" + + case "$BUILD_STATUS" in + running) + BITBUCKET_STATE="INPROGRESS" + BITBUCKET_DESCRIPTION="The build is running!" + ;; + passed) + BITBUCKET_STATE="SUCCESSFUL" + BITBUCKET_DESCRIPTION="The build passed!" + ;; + failed) + BITBUCKET_STATE="FAILED" + BITBUCKET_DESCRIPTION="The build failed." + ;; + esac + + echo "Pushing status to $BITBUCKET_STATUS_API..." + curl --request POST $BITBUCKET_STATUS_API \ + --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ + --header "Content-Type:application/json" \ + --silent \ + --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": + \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" + ``` 1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push pipeline success and failures to Bitbucket. - ```yaml - stages: - - test - - ci_status - - unit-tests: - script: - - echo "Success. Add your tests!" - - success: - stage: ci_status - before_script: - - "" - after_script: - - "" - script: - - BUILD_STATUS=passed BUILD_KEY=push ./build_status - when: on_success - - failure: - stage: ci_status - before_script: - - "" - after_script: - - "" - script: - - BUILD_STATUS=failed BUILD_KEY=push ./build_status - when: on_failure - ``` + ```yaml + stages: + - test + - ci_status + + unit-tests: + script: + - echo "Success. Add your tests!" + + success: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=passed BUILD_KEY=push ./build_status + when: on_success + + failure: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=failed BUILD_KEY=push ./build_status + when: on_failure + ``` GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines configured in `.gitlab-ci.yml` and push the status to Bitbucket. diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 0bb3aa35ed0..f639e3dadee 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -19,12 +19,12 @@ administrator: 1. In GitLab create a **CI/CD for external repo** project and select **GitHub**. -  +  1. Once authenticated, you will be redirected to a list of your repositories to connect. Click **Connect** to select the repository. -  +  1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). @@ -55,14 +55,14 @@ repositories: Token**. This token with be used to access your repository and push commit statuses to GitHub. - The `repo` and `admin:repo_hook` should be enable to allow GitLab access to - your project, update commit statuses, and create a web hook to notify - GitLab of new commits. + The `repo` and `admin:repo_hook` should be enable to allow GitLab access to + your project, update commit statuses, and create a web hook to notify + GitLab of new commits. 1. In GitLab create a **CI/CD for external repo** project and select **GitHub**. -  +  1. Paste the token into the **Personal access token** field and click **List Repositories**. Click **Connect** to select the repository. @@ -86,21 +86,21 @@ your repository: Access Token.** GitLab will use this token to access your repository and push commit statuses. - Enter a **Token description** and update the scope to allow: + Enter a **Token description** and update the scope to allow: - `repo` so that GitLab can access your project and update commit statuses + `repo` so that GitLab can access your project and update commit statuses 1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS URL for your GitHub repository. If your project is private, use the personal access token you just created for authentication. - GitLab will automatically configure polling-based pull mirroring. + GitLab will automatically configure polling-based pull mirroring. 1. Still in GitLab, enable the [GitHub project integration](../../user/project/integrations/github.md) from **Settings > Integrations.** - Check the **Active** checkbox to enable the integration, paste your - personal access token and HTTPS repository URL into the form, and **Save.** + Check the **Active** checkbox to enable the integration, paste your + personal access token and HTTPS repository URL into the form, and **Save.** 1. Still in GitLab create a **Personal Access Token** with `API` scope to authenticate the GitHub web hook notifying GitLab of new commits. @@ -108,15 +108,15 @@ your repository: 1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of new commits. - The web hook URL should be set to the GitLab API to - [trigger pull mirroring](../../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), - using the GitLab personal access token we just created. + The web hook URL should be set to the GitLab API to + [trigger pull mirroring](../../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), + using the GitLab personal access token we just created. - ``` - https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> - ``` + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` -  +  1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md new file mode 100644 index 00000000000..1c38c08b7cb --- /dev/null +++ b/doc/ci/directed_acyclic_graph/index.md @@ -0,0 +1,76 @@ +--- +type: reference +--- + +# Directed Acyclic Graph + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47063) in GitLab 12.2 (enabled by `ci_dag_support` feature flag). + +A [directed acyclic graph](https://www.techopedia.com/definition/5739/directed-acyclic-graph-dag) can be +used in the context of a CI/CD pipeline to build relationships between jobs such that +execution is performed in the quickest possible manner, regardless how stages may +be set up. + +For example, you may have a specific tool or separate website that is built +as part of your main project. Using a DAG, you can specify the relationship between +these jobs and GitLab will then execute the jobs as soon as possible instead of waiting +for each stage to complete. + +Unlike other DAG solutions for CI/CD, GitLab does not require you to choose one or the +other. You can implement a hybrid combination of DAG and traditional +stage-based operation within a single pipeline. Configuration is kept very simple, +requiring a single keyword to enable the feature for any job. + +Consider a monorepo as follows: + +``` +./service_a +./service_b +./service_c +./service_d +``` + +It has a pipeline that looks like the following: + +| build | test | deploy | +| ----- | ---- | ------ | +| build_a | test_a | deploy_a | +| build_b | test_b | deploy_b | +| build_c | test_c | deploy_c | +| build_d | test_d | deploy_d | + +Using a DAG, you can relate the `_a` jobs to each other separately from the `_b` jobs, +and even if service `a` takes a very long time to build, service `b` will not +wait for it and will finish as quickly as it can. In this very same pipeline, `_c` and +`_d` can be left alone and will run together in staged sequence just like any normal +GitLab pipeline. + +## Use cases + +A DAG can help solve several different kinds of relationships between jobs within +a CI/CD pipeline. Most typically this would cover when jobs need to fan in or out, +and/or merge back together (diamond dependencies). This can happen when you're +handling multi-platform builds or complex webs of dependencies as in something like +an operating system build or a complex deployment graph of independently deployable +but related microservices. + +Additionally, a DAG can help with general speediness of pipelines and helping +to deliver fast feedback. By creating dependency relationships that don't unnecessarily +block each other, your pipelines will run as quickly as possible regardless of +pipeline stages, ensuring output (including errors) is available to developers +as quickly as possible. + +## Usage + +Relationships are defined between jobs using the [`needs:` keyword](../yaml/README.md#needs). + +Note that `needs:` also works with the [parallel](../yaml/README.md#parallel) keyword, +giving you powerful options for parallelization within your pipeline. + +## Limitations + +A directed acyclic graph is a complicated feature, and as of the initial MVC there +are certain use cases that you may need to work around. For more information: + +- [`needs` requirements and limitations](../yaml/README.md#requirements-and-limitations). +- Related epic [gitlab-org#1716](https://gitlab.com/groups/gitlab-org/-/epics/1716). diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index efdcaf5a6f5..2cbad5f101c 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -6,7 +6,6 @@ type: concepts, howto GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects. - One of the new trends in Continuous Integration/Deployment is to: 1. Create an application image. @@ -29,7 +28,16 @@ during jobs. ## Runner Configuration -There are three methods to enable the use of `docker build` and `docker run` during jobs; each with their own tradeoffs. +There are three methods to enable the use of `docker build` and `docker run` +during jobs; each with their own tradeoffs. + +An alternative to using `docker build` is to [use kaniko](using_kaniko.md). +This avoids having to execute Runner in privileged mode. + +TIP: **Tip:** +To see how Docker and Runner are configured for shared Runners on +GitLab.com, see [GitLab.com Shared +Runners](../../user/gitlab_com/index.md#shared-runners). ### Use shell executor @@ -40,42 +48,42 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. 1. During GitLab Runner installation select `shell` as method of executing job scripts or use command: - ```bash - sudo gitlab-runner register -n \ - --url https://gitlab.com/ \ - --registration-token REGISTRATION_TOKEN \ - --executor shell \ - --description "My Runner" - ``` + ```bash + sudo gitlab-runner register -n \ + --url https://gitlab.com/ \ + --registration-token REGISTRATION_TOKEN \ + --executor shell \ + --description "My Runner" + ``` 1. Install Docker Engine on server. - For more information how to install Docker Engine on different systems - checkout the [Supported installations](https://docs.docker.com/engine/installation/). + For more information how to install Docker Engine on different systems + checkout the [Supported installations](https://docs.docker.com/engine/installation/). 1. Add `gitlab-runner` user to `docker` group: - ```bash - sudo usermod -aG docker gitlab-runner - ``` + ```bash + sudo usermod -aG docker gitlab-runner + ``` 1. Verify that `gitlab-runner` has access to Docker: - ```bash - sudo -u gitlab-runner -H docker info - ``` + ```bash + sudo -u gitlab-runner -H docker info + ``` - You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: + You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: - ```yaml - before_script: - - docker info + ```yaml + before_script: + - docker info - build_image: - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests - ``` + build_image: + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` 1. You can now use `docker` command (and **install** `docker-compose` if needed). @@ -83,101 +91,28 @@ NOTE: **Note:** By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. For more information please read [On Docker security: `docker` group considered harmful](https://www.andreas-jung.com/contents/on-docker-security-docker-group-considered-harmful). -### Use docker-in-docker executor +### Use docker-in-docker workflow with Docker executor The second approach is to use the special docker-in-docker (dind) [Docker image](https://hub.docker.com/_/docker/) with all tools installed (`docker`) and run the job script in context of that -image in privileged mode. - -NOTE: **Note:** `docker-compose` is not part of docker-in-docker (dind). In case you'd like to use `docker-compose` in your CI builds, please follow the [installation instructions for docker-compose](https://docs.docker.com/compose/install/) provided by docker. - -In order to do that, follow the steps: +image in privileged mode. -1. Install [GitLab Runner](https://docs.gitlab.com/runner/install). - -1. Register GitLab Runner from the command line to use `docker` and `privileged` - mode: - - ```bash - sudo gitlab-runner register -n \ - --url https://gitlab.com/ \ - --registration-token REGISTRATION_TOKEN \ - --executor docker \ - --description "My Docker Runner" \ - --docker-image "docker:stable" \ - --docker-privileged - ``` - - The above command will register a new Runner to use the special - `docker:stable` image which is provided by Docker. **Notice that it's using - the `privileged` mode to start the build and service containers.** If you - want to use [docker-in-docker] mode, you always have to use `privileged = true` - in your Docker containers. - - The above command will create a `config.toml` entry similar to this: - - ```toml - [[runners]] - url = "https://gitlab.com/" - token = TOKEN - executor = "docker" - [runners.docker] - tls_verify = false - image = "docker:stable" - privileged = true - disable_cache = false - volumes = ["/cache"] - [runners.cache] - Insecure = false - ``` - -1. You can now use `docker` in the build script (note the inclusion of the - `docker:dind` service): - - ```yaml - image: docker:stable +NOTE: **Note:** +`docker-compose` is not part of docker-in-docker (dind). To use `docker-compose` in your +CI builds, follow the `docker-compose` +[installation instructions](https://docs.docker.com/compose/install/). - variables: - # When using dind service we need to instruct docker, to talk with the - # daemon started inside of the service. The daemon is available with - # a network connection instead of the default /var/run/docker.sock socket. - # - # The 'docker' hostname is the alias of the service container as described at - # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services - # - # Note that if you're using the Kubernetes executor, the variable should be set to - # tcp://localhost:2375/ because of how the Kubernetes executor connects services - # to the job container - # DOCKER_HOST: tcp://localhost:2375/ - # - # For non-Kubernetes executors, we use tcp://docker:2375/ - DOCKER_HOST: tcp://docker:2375/ - # When using dind, it's wise to use the overlayfs driver for - # improved performance. - DOCKER_DRIVER: overlay2 - - services: - - docker:dind - - before_script: - - docker info - - build: - stage: build - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests - ``` +DANGER: **Danger:** +By enabling `--docker-privileged`, you are effectively disabling all of +the security mechanisms of containers and exposing your host to privilege +escalation which can lead to container breakout. For more information, check +out the official Docker documentation on +[Runtime privilege and Linux capabilities][docker-cap]. Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges: -- By enabling `--docker-privileged`, you are effectively disabling all of - the security mechanisms of containers and exposing your host to privilege - escalation which can lead to container breakout. For more information, check - out the official Docker documentation on - [Runtime privilege and Linux capabilities][docker-cap]. - When using docker-in-docker, each job is in a clean environment without the past history. Concurrent jobs work fine because every build gets it's own instance of Docker engine so they won't conflict with each other. But this @@ -187,7 +122,7 @@ not without its own challenges: [Using the overlayfs driver](#using-the-overlayfs-driver). - Since the `docker:dind` container and the runner container don't share their root filesystem, the job's working directory can be used as a mount point for - children containers. For example, if you have files you want to share with a + child containers. For example, if you have files you want to share with a child container, you may create a subdirectory under `/builds/$CI_PROJECT_PATH` and use it as your mount point (for a more thorough explanation, check [issue #41227](https://gitlab.com/gitlab-org/gitlab-ce/issues/41227)): @@ -203,6 +138,177 @@ not without its own challenges: An example project using this approach can be found here: <https://gitlab.com/gitlab-examples/docker>. +In the examples below, we are using Docker images tags to specify a +specific version, such as `docker:19.03.1`. If tags like `docker:stable` +are used, you have no control over what version is going to be used and this +can lead to unpredictable behavior, especially when new versions are +released. + +#### TLS enabled + +NOTE: **Note** +This requires GitLab Runner 11.11 or higher. + +The Docker daemon supports connection over TLS and it's done by default +for Docker 19.03.1 or higher. This is the **suggested** way to use the +docker-in-docker service and +[GitLab.com Shared Runners](../../user/gitlab_com/index.html#shared-runners) +support this. + +1. Install [GitLab Runner](https://docs.gitlab.com/runner/install). + +1. Register GitLab Runner from the command line to use `docker` and `privileged` + mode: + + ```bash + sudo gitlab-runner register -n \ + --url https://gitlab.com/ \ + --registration-token REGISTRATION_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "docker:19.03.1" \ + --docker-privileged \ + --docker-volumes "/certs/client" + ``` + + The above command will register a new Runner to use the special + `docker:19.03.1` image, which is provided by Docker. **Notice that it's + using the `privileged` mode to start the build and service + containers.** If you want to use [docker-in-docker] mode, you always + have to use `privileged = true` in your Docker containers. + + This will also mount `/certs/client` for the service and build + container, which is needed for the docker client to use the + certificates inside of that directory. For more information how + Docker with TLS works check <https://hub.docker.com/_/docker/#tls>. + + The above command will create a `config.toml` entry similar to this: + + ```toml + [[runners]] + url = "https://gitlab.com/" + token = TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:19.03.1" + privileged = true + disable_cache = false + volumes = ["/certs/client", "/cache"] + [runners.cache] + [runners.cache.s3] + [runners.cache.gcs] + ``` + +1. You can now use `docker` in the build script (note the inclusion of the + `docker:19.03.1-dind` service): + + ```yaml + image: docker:19.03.1 + + variables: + # When using dind service, we need to instruct docker, to talk with + # the daemon started inside of the service. The daemon is available + # with a network connection instead of the default + # /var/run/docker.sock socket. docker:19.03.1 does this automatically + # by setting the DOCKER_HOST in + # https://github.com/docker-library/docker/blob/d45051476babc297257df490d22cbd806f1b11e4/19.03.1/docker-entrypoint.sh#L23-L29 + # + # The 'docker' hostname is the alias of the service container as described at + # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services. + # + # Note that if you're using the Kubernetes executor, the variable + # should be set to tcp://localhost:2376/ because of how the + # Kubernetes executor connects services to the job container + # DOCKER_HOST: tcp://localhost:2376/ + # + # When using dind, it's wise to use the overlayfs driver for + # improved performance. + DOCKER_DRIVER: overlay2 + # Specify to Docker where to create the certificates, Docker will + # create them automatically on boot, and will create + # `/certs/client` that will be shared between the service and job + # container, thanks to volume mount from config.toml + DOCKER_TLS_CERTDIR: "/certs" + + services: + - docker:19.03.1-dind + + before_script: + - docker info + + build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` + +#### TLS disabled + +Sometimes there are legitimate reasons why you might want to disable TLS. +For example, you have no control over the GitLab Runner configuration +that you are using. + +Assuming that the Runner `config.toml` is similar to: + +```toml +[[runners]] + url = "https://gitlab.com/" + token = TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:19.03.1" + privileged = true + disable_cache = false + volumes = ["/cache"] + [runners.cache] + [runners.cache.s3] + [runners.cache.gcs] +``` + +You can now use `docker` in the build script (note the inclusion of the +`docker:19.03.1-dind` service): + +```yaml +image: docker:19.03.1 + +variables: + # When using dind service we need to instruct docker, to talk with the + # daemon started inside of the service. The daemon is available with + # a network connection instead of the default /var/run/docker.sock socket. + # + # The 'docker' hostname is the alias of the service container as described at + # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services + # + # Note that if you're using the Kubernetes executor, the variable should be set to + # tcp://localhost:2375/ because of how the Kubernetes executor connects services + # to the job container + # DOCKER_HOST: tcp://localhost:2375/ + # + # For non-Kubernetes executors, we use tcp://docker:2375/ + DOCKER_HOST: tcp://docker:2375/ + # When using dind, it's wise to use the overlayfs driver for + # improved performance. + DOCKER_DRIVER: overlay2 + # + # This will instruct Docker not to start over TLS. + DOCKER_TLS_CERTDIR: "" + +services: + - docker:19.03.1-dind + +before_script: + - docker info + +build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests +``` + ### Use Docker socket binding The third approach is to bind-mount `/var/run/docker.sock` into the @@ -220,54 +326,54 @@ In order to do that, follow the steps: 1. Register GitLab Runner from the command line to use `docker` and share `/var/run/docker.sock`: - ```bash - sudo gitlab-runner register -n \ - --url https://gitlab.com/ \ - --registration-token REGISTRATION_TOKEN \ - --executor docker \ - --description "My Docker Runner" \ - --docker-image "docker:stable" \ - --docker-volumes /var/run/docker.sock:/var/run/docker.sock - ``` - - The above command will register a new Runner to use the special - `docker:stable` image which is provided by Docker. **Notice that it's using - the Docker daemon of the Runner itself, and any containers spawned by docker - commands will be siblings of the Runner rather than children of the runner.** - This may have complications and limitations that are unsuitable for your workflow. - - The above command will create a `config.toml` entry similar to this: - - ```toml - [[runners]] - url = "https://gitlab.com/" - token = REGISTRATION_TOKEN - executor = "docker" - [runners.docker] - tls_verify = false - image = "docker:stable" - privileged = false - disable_cache = false - volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] - [runners.cache] - Insecure = false - ``` + ```bash + sudo gitlab-runner register -n \ + --url https://gitlab.com/ \ + --registration-token REGISTRATION_TOKEN \ + --executor docker \ + --description "My Docker Runner" \ + --docker-image "docker:stable" \ + --docker-volumes /var/run/docker.sock:/var/run/docker.sock + ``` + + The above command will register a new Runner to use the special + `docker:stable` image which is provided by Docker. **Notice that it's using + the Docker daemon of the Runner itself, and any containers spawned by docker + commands will be siblings of the Runner rather than children of the runner.** + This may have complications and limitations that are unsuitable for your workflow. + + The above command will create a `config.toml` entry similar to this: + + ```toml + [[runners]] + url = "https://gitlab.com/" + token = REGISTRATION_TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:stable" + privileged = false + disable_cache = false + volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] + [runners.cache] + Insecure = false + ``` 1. You can now use `docker` in the build script (note that you don't need to include the `docker:dind` service as when using the Docker in Docker executor): - ```yaml - image: docker:stable + ```yaml + image: docker:stable - before_script: - - docker info + before_script: + - docker info - build: - stage: build - script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests - ``` + build: + stage: build + script: + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests + ``` While the above method avoids using Docker in privileged mode, you should be aware of the following implications: @@ -283,9 +389,9 @@ aware of the following implications: work as expected since volume mounting is done in the context of the host machine, not the build container. For example: - ```sh - docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests - ``` + ```sh + docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests + ``` ## Making docker-in-docker builds faster with Docker layer caching @@ -356,23 +462,23 @@ which can be avoided if a different driver is used, for example `overlay2`. 1. Make sure a recent kernel is used, preferably `>= 4.2`. 1. Check whether the `overlay` module is loaded: - ```sh - sudo lsmod | grep overlay - ``` + ```sh + sudo lsmod | grep overlay + ``` - If you see no result, then it isn't loaded. To load it use: + If you see no result, then it isn't loaded. To load it use: - ```sh - sudo modprobe overlay - ``` + ```sh + sudo modprobe overlay + ``` - If everything went fine, you need to make sure module is loaded on reboot. - On Ubuntu systems, this is done by editing `/etc/modules`. Just add the - following line into it: + If everything went fine, you need to make sure module is loaded on reboot. + On Ubuntu systems, this is done by editing `/etc/modules`. Just add the + following line into it: - ```text - overlay - ``` + ```text + overlay + ``` ### Use driver per project @@ -440,9 +546,9 @@ For all projects, mostly suitable for public ones: your Docker images and has read/write access to the Registry. This is ephemeral, so it's only valid for one job. You can use the following example as-is: - ```sh - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - ``` + ```sh + docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + ``` For private and internal projects: @@ -455,9 +561,9 @@ For private and internal projects: Replace the `<username>` and `<access_token>` in the following example: - ```sh - docker login -u <username> -p <access_token> $CI_REGISTRY - ``` + ```sh + docker login -u <username> -p <access_token> $CI_REGISTRY + ``` - **Using the GitLab Deploy Token**: You can create and use a [special deploy token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token) @@ -465,9 +571,9 @@ For private and internal projects: Once created, you can use the special environment variables, and GitLab CI/CD will fill them in for you. You can use the following example as-is: - ```sh - docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY - ``` + ```sh + docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY + ``` ### Container Registry examples diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 2d7fb323d79..489791141ed 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -35,8 +35,8 @@ sudo gitlab-runner register \ --description "docker-ruby-2.1" \ --executor "docker" \ --docker-image ruby:2.1 \ - --docker-postgres latest \ - --docker-mysql latest + --docker-services postgres:latest \ + --docker-services mysql:latest ``` The registered runner will use the `ruby:2.1` Docker image and will run two @@ -193,13 +193,14 @@ You can simply define an image that will be used for all jobs and a list of services that you want to use during build time: ```yaml -image: ruby:2.2 +default: + image: ruby:2.2 -services: - - postgres:9.3 + services: + - postgres:9.3 -before_script: - - bundle install + before_script: + - bundle install test: script: @@ -209,8 +210,9 @@ test: It is also possible to define different images and services per job: ```yaml -before_script: - - bundle install +default: + before_script: + - bundle install test:2.1: image: ruby:2.1 @@ -231,16 +233,53 @@ Or you can pass some [extended configuration options](#extended-docker-configura for `image` and `services`: ```yaml +default: + image: + name: ruby:2.2 + entrypoint: ["/bin/bash"] + + services: + - name: my-postgres:9.4 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] + + before_script: + - bundle install + +test: + script: + - bundle exec rake spec +``` + +## Passing environment variables to services + +You can also pass custom environment [variables](../variables/README.md) +to fine tune your Docker `images` and `services` directly in the `.gitlab-ci.yml` file. +For more information, see [custom environment variables](../variables/README.md#gitlab-ciyml-defined-variables) + +```yaml +# The following variables will automatically be passed down to the Postgres container +# as well as the Ruby container and available within each. +variables: + HTTPS_PROXY: "https://10.1.1.1:8090" + HTTP_PROXY: "https://10.1.1.1:8090" + POSTGRES_DB: "my_custom_db" + POSTGRES_USER: "postgres" + POSTGRES_PASSWORD: "example" + PGDATA: "/var/lib/postgresql/data" + POSTGRES_INITDB_ARGS: "--encoding=UTF8 --data-checksums" + +services: +- name: postgres:9.4 + alias: db + entrypoint: ["docker-entrypoint.sh"] + command: ["postgres"] + image: name: ruby:2.2 entrypoint: ["/bin/bash"] -services: -- name: my-postgres:9.4 - alias: db-postgres - entrypoint: ["/usr/local/bin/db-postgres"] - command: ["start"] - before_script: - bundle install @@ -266,25 +305,25 @@ For example, the following two definitions are equal: 1. Using a string as an option to `image` and `services`: - ```yaml - image: "registry.example.com/my/image:latest" + ```yaml + image: "registry.example.com/my/image:latest" - services: - - postgresql:9.4 - - redis:latest - ``` + services: + - postgresql:9.4 + - redis:latest + ``` 1. Using a map as an option to `image` and `services`. The use of `image:name` is required: - ```yaml - image: - name: "registry.example.com/my/image:latest" + ```yaml + image: + name: "registry.example.com/my/image:latest" - services: - - name: postgresql:9.4 - - name: redis:latest - ``` + services: + - name: postgresql:9.4 + - name: redis:latest + ``` ### Available settings for `image` @@ -456,14 +495,6 @@ that runner. ## Define an image from a private Container Registry -> **Notes:** -> -> - This feature requires GitLab Runner **1.8** or higher -> - For GitLab Runner versions **>= 0.6, <1.8** there was a partial -> support for using private registries, which required manual configuration -> of credentials on runner's host. We recommend to upgrade your Runner to -> at least version **1.8** if you want to use private registries. - To access private container registries, the GitLab Runner process can use: - [Statically defined credentials](#using-statically-defined-credentials). That is, a username and password for a specific registry. @@ -486,7 +517,19 @@ it's provided as an environment variable. This is because GitLab Runnner uses ** `config.toml` configuration and doesn't interpolate **ANY** environment variables at runtime. +### Requirements and limitations + +- This feature requires GitLab Runner **1.8** or higher. +- For GitLab Runner versions **>= 0.6, <1.8** there was a partial + support for using private registries, which required manual configuration + of credentials on runner's host. We recommend to upgrade your Runner to + at least version **1.8** if you want to use private registries. +- Not available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html), + follow <https://gitlab.com/gitlab-org/gitlab-runner/issues/2673> for + details. + ### Using statically-defined credentials + There are two approaches that you can take in order to access a private registry. Both require setting the environment variable `DOCKER_AUTH_CONFIG` with appropriate authentication info. @@ -516,18 +559,18 @@ There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: - **First way -** Do a `docker login` on your local machine: - ```bash - docker login registry.example.com:5000 --username my_username --password my_password - ``` + ```bash + docker login registry.example.com:5000 --username my_username --password my_password + ``` - Then copy the content of `~/.docker/config.json`. + Then copy the content of `~/.docker/config.json`. - If you don't need access to the registry from your computer, you - can do a `docker logout`: + If you don't need access to the registry from your computer, you + can do a `docker logout`: - ```bash - docker logout registry.example.com:5000 - ``` + ```bash + docker logout registry.example.com:5000 + ``` - **Second way -** In some setups, it's possible that Docker client will use the available system keystore to store the result of `docker @@ -536,12 +579,12 @@ There are two ways to determine the value of `DOCKER_AUTH_CONFIG`: `${username}:${password}` manually. Open a terminal and execute the following command: - ```bash - echo -n "my_username:my_password" | base64 + ```bash + echo -n "my_username:my_password" | base64 - # Example output to copy - bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= - ``` + # Example output to copy + bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= + ``` #### Configuring a job @@ -551,25 +594,25 @@ follow these steps: 1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: - ```json - { - "auths": { - "registry.example.com:5000": { - "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=" - } - } - } - ``` + ```json + { + "auths": { + "registry.example.com:5000": { + "auth": "bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=" + } + } + } + ``` 1. You can now use any private image from `registry.example.com:5000` defined in `image` and/or `services` in your `.gitlab-ci.yml` file: - ```yaml - image: registry.example.com:5000/namespace/image:tag - ``` + ```yaml + image: registry.example.com:5000/namespace/image:tag + ``` - In the example above, GitLab Runner will look at `registry.example.com:5000` for the - image `namespace/image:tag`. + In the example above, GitLab Runner will look at `registry.example.com:5000` for the + image `namespace/image:tag`. You can add configuration for as many registries as you want, adding more registries to the `"auths"` hash as described above. @@ -598,10 +641,10 @@ To add `DOCKER_AUTH_CONFIG` to a Runner: 1. Modify the Runner's `config.toml` file as follows: - ```toml - [[runners]] - environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] - ``` + ```toml + [[runners]] + environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com:5000\":{\"auth\":\"bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ=\"}}}"] + ``` 1. Restart the Runner service. @@ -623,16 +666,17 @@ To configure credentials store, follow these steps: Make sure helper program is available in GitLab Runner `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: + - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the - Docker configuration file as the value: + Docker configuration file as the value: - ```json - { - "credsStore": "osxkeychain" - } - ``` + ```json + { + "credsStore": "osxkeychain" + } + ``` - Or, if you are running self-hosted Runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. GitLab Runner will read this config file @@ -654,17 +698,18 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th 1. Make sure `docker-credential-ecr-login` is available in GitLab Runner's `$PATH`. 1. Make GitLab Runner use it. There are two ways to accomplish this. Either: + - Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the - Docker configuration file as the value: + Docker configuration file as the value: - ```json - { - "credHelpers": { - "aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login" - } - } - ``` + ```json + { + "credHelpers": { + "aws_account_id.dkr.ecr.region.amazonaws.com": "ecr-login" + } + } + ``` - Or, if you are running self-hosted Runners, add the above JSON to `${GITLAB_RUNNER_HOME}/.docker/config.json`. @@ -674,12 +719,12 @@ To configure access for `aws_account_id.dkr.ecr.region.amazonaws.com`, follow th 1. You can now use any private image from `aws_account_id.dkr.ecr.region.amazonaws.com` defined in `image` and/or `services` in your `.gitlab-ci.yml` file: - ```yaml - image: aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest - ``` + ```yaml + image: aws_account_id.dkr.ecr.region.amazonaws.com/private/image:latest + ``` - In the example above, GitLab Runner will look at `aws_account_id.dkr.ecr.region.amazonaws.com` for the - image `private/image:latest`. + In the example above, GitLab Runner will look at `aws_account_id.dkr.ecr.region.amazonaws.com` for the + image `private/image:latest`. You can add configuration for as many registries as you want, adding more registries to the `"credHelpers"` hash as described above. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 50f1ac3d54a..925653f9fdf 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -11,7 +11,8 @@ Requires GitLab Runner 11.2 and above. container images from a Dockerfile, inside a container or Kubernetes cluster. kaniko solves two problems with using the -[docker-in-docker build](using_docker_build.md#use-docker-in-docker-executor) method: +[docker-in-docker +build](using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor) method: - Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) in order to function, which is a significant security concern. diff --git a/doc/ci/environments.md b/doc/ci/environments.md index f86ca8f74f2..f6c47a99712 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -591,15 +591,13 @@ exist, you should see something like: ### Monitoring environments -> **Notes:** -> -> - For the monitoring dashboard to appear, you need to: -> - Enable the [Prometheus integration](../user/project/integrations/prometheus.md). -> - Configure Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/index.md). -> - With GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. - If you have enabled [Prometheus for monitoring system and response metrics](../user/project/integrations/prometheus.md), -you can monitor the behavior of your app running in each environment. +you can monitor the behavior of your app running in each environment. For the monitoring +dashboard to appear, you need to Configure Prometheus to collect at least one +[supported metric](../user/project/integrations/prometheus_library/index.md). + +NOTE: **Note:** +Since GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. Once configured, GitLab will attempt to retrieve [supported performance metrics](../user/project/integrations/prometheus_library/index.md) for any environment that has had a successful deployment. If monitoring data was @@ -621,6 +619,10 @@ versions of the app, all without leaving GitLab. Add a [button to the Monitoring dashboard](../user/project/operations/linking_to_an_external_dashboard.md) linking directly to your existing external dashboards. +#### Embedding metrics in GitLab Flavored Markdown + +Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab Flavored Markdown](../user/project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown) for more details. + ### Web terminals > Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. @@ -692,7 +694,7 @@ with `review/` would have that particular variable. Some GitLab features can behave differently for each environment. For example, you can -[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables-premium). **(PREMIUM)** +[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables). In most cases, these features use the _environment specs_ mechanism, which offers an efficient way to implement scoping within each environment group. diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 5a302392c54..00995f881da 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -33,10 +33,11 @@ The following table lists examples with step-by-step tutorials that are containe | Java with Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | | PHP with PHPunit, atoum | [Testing PHP projects](php.md). | | PHP with NPM, SCP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP with Laravel, Ennvoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | | Python on Heroku | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | | Ruby on Heroku | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | | Scala on Heroku | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Parallel testing Ruby & JS | [GitLab CI parallel jobs testing for Ruby & JavaScript projects](https://docs.knapsackpro.com/2019/how-to-run-parallel-jobs-for-rspec-tests-on-gitlab-ci-pipeline-and-speed-up-ruby-javascript-testing). | ### Contributing examples diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index c9f700ed190..e5f307e570d 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -1,5 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' +disqus_identifier: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' author: Fabio Busatto author_gitlab: bikebilly level: intermediate @@ -39,9 +39,10 @@ project: 1. Create a new project by selecting **Import project from ➔ Repo by URL** 1. Add the following URL: - ``` - https://gitlab.com/gitlab-examples/maven/simple-maven-dep.git - ``` + ``` + https://gitlab.com/gitlab-examples/maven/simple-maven-dep.git + ``` + 1. Click **Create project** This application is nothing more than a basic class with a stub for a JUnit based test suite. @@ -63,15 +64,15 @@ The application is ready to use, but you need some additional steps to deploy it 1. Copy the snippet in the `pom.xml` file for your project, just after the `dependencies` section. The snippet should look like this: - ```xml - <distributionManagement> - <repository> - <id>central</id> - <name>83d43b5afeb5-releases</name> - <url>${env.MAVEN_REPO_URL}/libs-release-local</url> - </repository> - </distributionManagement> - ``` + ```xml + <distributionManagement> + <repository> + <id>central</id> + <name>83d43b5afeb5-releases</name> + <url>${env.MAVEN_REPO_URL}/libs-release-local</url> + </repository> + </distributionManagement> + ``` Another step you need to do before you can deploy the dependency to Artifactory is to configure the authentication data. It is a simple task, but Maven requires @@ -86,18 +87,18 @@ parameter in `.gitlab-ci.yml` to use the custom location instead of the default 1. Create a file called `settings.xml` in the `.m2` folder 1. Copy the following content into a `settings.xml` file: - ```xml - <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" - xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> - <servers> - <server> - <id>central</id> - <username>${env.MAVEN_REPO_USER}</username> - <password>${env.MAVEN_REPO_PASS}</password> - </server> - </servers> - </settings> - ``` + ```xml + <settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" + xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <servers> + <server> + <id>central</id> + <username>${env.MAVEN_REPO_USER}</username> + <password>${env.MAVEN_REPO_PASS}</password> + </server> + </servers> + </settings> + ``` Username and password will be replaced by the correct values using variables. @@ -187,9 +188,10 @@ We'll use again a Maven app that can be cloned from our example project: 1. Create a new project by selecting **Import project from ➔ Repo by URL** 1. Add the following URL: - ``` - https://gitlab.com/gitlab-examples/maven/simple-maven-app.git - ``` + ``` + https://gitlab.com/gitlab-examples/maven/simple-maven-app.git + ``` + 1. Click **Create project** This one is a simple app as well. If you look at the `src/main/java/com/example/app/App.java` @@ -204,13 +206,13 @@ Since Maven doesn't know how to resolve the dependency, you need to modify the c 1. Copy the snippet in the `dependencies` section of the `pom.xml` file. The snippet should look like this: - ```xml - <dependency> - <groupId>com.example.dep</groupId> - <artifactId>simple-maven-dep</artifactId> - <version>1.0</version> - </dependency> - ``` + ```xml + <dependency> + <groupId>com.example.dep</groupId> + <artifactId>simple-maven-dep</artifactId> + <version>1.0</version> + </dependency> + ``` ### Configure the Artifactory repository location diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index 8ecac4a5a4f..3266e5dc62e 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -15,7 +15,7 @@ your code by using GitLab CI/CD and [sitespeed.io](https://www.sitespeed.io) using Docker-in-Docker. First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +[docker-in-docker build](../docker/using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor). Once you set up the Runner, add a new job to `.gitlab-ci.yml` that generates the expected report: @@ -155,4 +155,4 @@ performance: paths: - performance.json - sitespeed-results/ -```
\ No newline at end of file +``` diff --git a/doc/ci/examples/code_quality.md b/doc/ci/examples/code_quality.md index 43f773dab7c..9c65de115b4 100644 --- a/doc/ci/examples/code_quality.md +++ b/doc/ci/examples/code_quality.md @@ -1,5 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/ci/examples/code_climate.html' +disqus_identifier: 'https://docs.gitlab.com/ee/ci/examples/code_climate.html' type: reference, howto --- @@ -14,7 +14,7 @@ This example shows how to run Code Quality on your code by using GitLab CI/CD and Docker. First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-workflow-with-docker-executor). Once you set up the Runner, include the CodeQuality template in your CI config: @@ -34,6 +34,12 @@ For [GitLab Starter][ee] users, this information will be automatically extracted and shown right in the merge request widget. [Learn more on Code Quality in merge requests](../../user/project/merge_requests/code_quality.md). +CAUTION: **Caution:** +On self-managed instances, if a malicious actor compromises the Code Quality job +definition they will be able to execute privileged docker commands on the Runner +host. Having proper access control policies mitigates this attack vector by +allowing access only to trusted actors. + ## Previous job definitions CAUTION: **Caution:** diff --git a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md index 538843ab8dc..1d4c9221cf2 100644 --- a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md +++ b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md @@ -79,10 +79,10 @@ image: java:8 stages: - build - deploy - + before_script: - chmod +x mvnw - + build: stage: build script: ./mvnw package diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 50e61cafeb9..44d3ec8046c 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -418,10 +418,14 @@ fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/lat 1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) 1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later -  + +  + 1. Go to your GitLab project, click **Settings > CI/CD** on the left sidebar 1. Expand the **Variables** section -  + +  + 1. Add a key named `AWS_KEY_ID` and copy the key id from Step 2 into the **Value** textbox 1. Add a key named `AWS_KEY_SECRET` and copy the key secret from Step 2 into the **Value** textbox diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png b/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png Binary files differindex c45d70d7f7a..9fe1739f37e 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png +++ b/doc/ci/examples/end_to_end_testing_webdriverio/img/deployed_dependency_update.png diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 38fcf05f519..38d0d86ffa2 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -143,7 +143,7 @@ Which brings us to the exciting part: how do we run this in GitLab CI/CD? There need to do for this: 1. Set up [CI/CD jobs](../../yaml/README.md#introduction) that actually have a browser available. -2. Update our WebdriverIO configuration to use those browsers to visit the review apps. +1. Update our WebdriverIO configuration to use those browsers to visit the review apps. For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/README.md#stages) `confidence-check` that is executed _after_ the stage that deploys the review app. It uses the `node:latest` [Docker diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index d7308a3a5ec..808e4285f2f 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -1,5 +1,5 @@ --- -redirect_from: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' +disqus_identifier: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' author: Mehran Rasulian author_gitlab: mehranrasulian level: intermediate @@ -452,6 +452,7 @@ To start using Container Registry on our machine, we first need to login to the ```bash docker login registry.gitlab.com ``` + Then we can build and push our image to GitLab: ```bash diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 0b9e9e93e55..1dd3049d53d 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -6,7 +6,7 @@ type: tutorial This guide covers basic building instructions for PHP projects. -Two testing scenarios are covered: using the Docker executor and +Two testing scenarios are covered: using the Docker executor and using the Shell executor. ## Test PHP projects using the Docker executor diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index a5fed00972f..0e595e1a0be 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -188,28 +188,27 @@ when running our Phoenix in our `localhost`. - Open `hello_gitlab_ci/config/test.exs` on your favorite code editor - Go to **Configure your database** session and edit the block to include `System.get_env`: - ```elixir - # Configure your database - config :hello_gitlab_ci, HelloGitlabCi.Repo, - adapter: Ecto.Adapters.Postgres, - username: System.get_env("POSTGRES_USER") || "postgres", - password: System.get_env("POSTGRES_PASSWORD") || "postgres", - database: System.get_env("POSTGRES_DB") || "hello_gitlab_ci_test", - hostname: System.get_env("POSTGRES_HOST") || "localhost", - pool: Ecto.Adapters.SQL.Sandbox - ``` - - We'll need these system variables later on. + ```elixir + # Configure your database + config :hello_gitlab_ci, HelloGitlabCi.Repo, + adapter: Ecto.Adapters.Postgres, + username: System.get_env("POSTGRES_USER") || "postgres", + password: System.get_env("POSTGRES_PASSWORD") || "postgres", + database: System.get_env("POSTGRES_DB") || "hello_gitlab_ci_test", + hostname: System.get_env("POSTGRES_HOST") || "localhost", + pool: Ecto.Adapters.SQL.Sandbox + ``` + + We'll need these system variables later on. - Create an empty file named `.gitkeep` into `hello_gitlab_ci/priv/repo/migrations` - As our project is still fresh, we don't have any data on our database, so, the `migrations` -directory will be empty. - Without `.gitkeep`, git will not upload this empty directory and we'll got an error when running our -test on GitLab. + As our project is still fresh, we don't have any data on our database, so, the `migrations` + directory will be empty. + Without `.gitkeep`, git will not upload this empty directory and we'll got an error when running our + test on GitLab. - > **Note:** - If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. + > **Note:** If we add a folder via the GitLab UI, GitLab itself will add the `.gitkeep` to that new dir. Now, let's run a local test and see if everything we did didn't break anything. @@ -248,64 +247,64 @@ project. - The fastest and easiest way to do this, is to click on **Set up CI** on project's main page: -  +  - On next screen, we can select a template ready to go. Click on **Apply a GitLab CI/CD Yaml template** and select **Elixir**: -  +  - This template file tells GitLab CI/CD about what we wish to do every time a new commit is made. - However, we have to adapt it to run a Phoenix app. + This template file tells GitLab CI/CD about what we wish to do every time a new commit is made. + However, we have to adapt it to run a Phoenix app. - The first line tells GitLab what Docker image will be used. - Remember when we learn about Runners, the isolated virtual machine where GitLab CI/CD build and test - our application? This virtual machine must have all dependencies to run our application. This is - where a Docker image is needed. The correct image will provide the entire system for us. + Remember when we learn about Runners, the isolated virtual machine where GitLab CI/CD build and test + our application? This virtual machine must have all dependencies to run our application. This is + where a Docker image is needed. The correct image will provide the entire system for us. - As a suggestion, you can use [trenpixster's elixir image][docker-image], which already has all - dependencies for Phoenix installed, such as Elixir, Erlang, NodeJS and PostgreSQL: + As a suggestion, you can use [trenpixster's elixir image][docker-image], which already has all + dependencies for Phoenix installed, such as Elixir, Erlang, NodeJS and PostgreSQL: - ```yml - image: trenpixster/elixir:latest - ``` + ```yml + image: trenpixster/elixir:latest + ``` - At `services` session, we'll only use `postgres`, so we'll delete `mysql` and `redis` lines: - ```yml - services: - - postgres:latest - ``` + ```yml + services: + - postgres:latest + ``` - Now, we'll create a new entry called `variables`, before `before_script` session: - ```yml - variables: - POSTGRES_DB: hello_gitlab_ci_test - POSTGRES_HOST: postgres - POSTGRES_USER: postgres - POSTGRES_PASSWORD: "postgres" - MIX_ENV: "test" - ``` + ```yml + variables: + POSTGRES_DB: hello_gitlab_ci_test + POSTGRES_HOST: postgres + POSTGRES_USER: postgres + POSTGRES_PASSWORD: "postgres" + MIX_ENV: "test" + ``` - Here, we are setting up the values for GitLab CI/CD authenticate into PostgreSQL, as we did on - `config/test.exs` earlier. + Here, we are setting up the values for GitLab CI/CD authenticate into PostgreSQL, as we did on + `config/test.exs` earlier. - In `before_script` session, we'll add some commands to prepare everything to the test: - ```yml - before_script: - - apt-get update && apt-get -y install postgresql-client - - mix local.hex --force - - mix deps.get --only test - - mix ecto.create - - mix ecto.migrate - ``` - - It's important to install `postgresql-client` to let GitLab CI/CD access PostgreSQL and create our - database with the login information provided earlier. More important is to respect the indentation, - to avoid syntax errors when running the build. + ```yml + before_script: + - apt-get update && apt-get -y install postgresql-client + - mix local.hex --force + - mix deps.get --only test + - mix ecto.create + - mix ecto.migrate + ``` + + It's important to install `postgresql-client` to let GitLab CI/CD access PostgreSQL and create our + database with the login information provided earlier. More important is to respect the indentation, + to avoid syntax errors when running the build. - And finally, we'll let `mix` session intact. diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 1354a26d6e2..cce33c7a6b4 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -69,12 +69,14 @@ correctly with your CI jobs: 1. Next, if you are using `gitlab-runner` v1.10+, you can set the `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive` to tell the runner to fetch your submodules before the job: - ```yaml - variables: - GIT_SUBMODULE_STRATEGY: recursive - ``` - See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy) - for more details about `GIT_SUBMODULE_STRATEGY`. + + ```yaml + variables: + GIT_SUBMODULE_STRATEGY: recursive + ``` + + See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy) + for more details about `GIT_SUBMODULE_STRATEGY`. 1. If you are using an older version of `gitlab-runner`, then use `git submodule sync/update` in `before_script`: diff --git a/doc/ci/img/collapsible_log.png b/doc/ci/img/collapsible_log.png Binary files differindex 2785033b349..d2a570e246e 100644 --- a/doc/ci/img/collapsible_log.png +++ b/doc/ci/img/collapsible_log.png diff --git a/doc/ci/img/deployments_view.png b/doc/ci/img/deployments_view.png Binary files differindex 12090434bef..9e2b7e89577 100644 --- a/doc/ci/img/deployments_view.png +++ b/doc/ci/img/deployments_view.png diff --git a/doc/ci/img/environments_available.png b/doc/ci/img/environments_available.png Binary files differindex 48fc6effc2d..6c64e9398f7 100644 --- a/doc/ci/img/environments_available.png +++ b/doc/ci/img/environments_available.png diff --git a/doc/ci/img/environments_mr_review_app.png b/doc/ci/img/environments_mr_review_app.png Binary files differindex 6a7b7ce5679..86c20d8d3b6 100644 --- a/doc/ci/img/environments_mr_review_app.png +++ b/doc/ci/img/environments_mr_review_app.png diff --git a/doc/ci/img/manual_job_variables.png b/doc/ci/img/manual_job_variables.png Binary files differnew file mode 100644 index 00000000000..a5ed351fdcd --- /dev/null +++ b/doc/ci/img/manual_job_variables.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 6c73cbbf8d7..58307660e51 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -8,7 +8,7 @@ type: reference Interactive web terminals give the user access to a terminal in GitLab for running one-off commands for their CI pipeline. Since this is giving the user -shell access to the environment where [GitLab Runner](https://docs.gitlab.com/runner/) +shell access to the environment where [GitLab Runner](https://docs.gitlab.com/runner/) is deployed, some [security precautions](../../administration/integration/terminal.md#security) were taken to protect the users. @@ -62,4 +62,3 @@ close the terminal window. ## Interactive Web Terminals for the Web IDE **(ULTIMATE ONLY)** Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md). - diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differindex 204e9c462e5..f3fb9444b55 100644 --- a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png +++ b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png diff --git a/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png b/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png Binary files differindex 5089a1088c5..a0874f66eaa 100644 --- a/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png +++ b/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index fc89f0fc94f..366aca3442e 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -9,6 +9,11 @@ In this document we'll present an overview of the concepts of Continuous Integra Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD. +NOTE: **Out-of-the-box management systems can decrease hours spent on maintaining toolchains by 10% or more.** +Watch our +["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to learn about continuous methods and how GitLab’s built-in CI can help you simplify and scale software development. + ## Introduction to CI/CD methodologies The continuous methodologies of software development are based on @@ -146,14 +151,14 @@ so, GitLab CI/CD: - Runs automated scripts (sequential or parallel) to: - Build and test your app. - Preview the changes per merge request with Review Apps, as you - would see in your `localhost`. + would see in your `localhost`. Once you're happy with your implementation: - Get your code reviewed and approved. - Merge the feature branch into the default branch. - GitLab CI/CD deploys your changes automatically to a production environment. -- And finally, you and your team can easily roll it back if something goes wrong. +- And finally, you and your team can easily roll it back if something goes wrong.  @@ -176,24 +181,24 @@ you'll see some of the features available in GitLab according to each stage (Verify, Package, Release). 1. **Verify**: - - Automatically build and test your application with Continuous Integration. - - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). **(STARTER)** - - Determine the performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **(PREMIUM)** - - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [JUnit tests](../junit_test_reports.md). - - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. + - Automatically build and test your application with Continuous Integration. + - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). **(STARTER)** + - Determine the performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **(PREMIUM)** + - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [JUnit tests](../junit_test_reports.md). + - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. 1. **Package**: - - Store Docker images with [Container Registry](../../user/project/container_registry.md). - - Store NPM packages with [NPM Registry](../../user/project/packages/npm_registry.md). **(PREMIUM)** - - Store Maven artifacts with [Maven Repository](../../user/project/packages/maven_repository.md). **(PREMIUM)** + - Store Docker images with [Container Registry](../../user/project/container_registry.md). + - Store NPM packages with [NPM Registry](../../user/project/packages/npm_registry.md). **(PREMIUM)** + - Store Maven artifacts with [Maven Repository](../../user/project/packages/maven_repository.md). **(PREMIUM)** 1. **Release**: - - Continuous Deployment, automatically deploying your app to production. - - Continuous Delivery, manually click to deploy your app to production. - - Deploy static websites with [GitLab Pages](../../user/project/pages/index.md). - - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](../../user/project/canary_deployments.md). **(PREMIUM)** - - Deploy your features behind [Feature Flags](../../user/project/operations/feature_flags.md). **(PREMIUM)** - - Add release notes to any Git tag with [GitLab Releases](../../user/project/releases/index.md). - - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](../../user/project/deploy_boards.md). **(PREMIUM)** - - Deploy your application to a production environment in a Kubernetes cluster with [Auto Deploy](../../topics/autodevops/index.md#auto-deploy). + - Continuous Deployment, automatically deploying your app to production. + - Continuous Delivery, manually click to deploy your app to production. + - Deploy static websites with [GitLab Pages](../../user/project/pages/index.md). + - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](../../user/project/canary_deployments.md). **(PREMIUM)** + - Deploy your features behind [Feature Flags](../../user/project/operations/feature_flags.md). **(PREMIUM)** + - Add release notes to any Git tag with [GitLab Releases](../../user/project/releases/index.md). + - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](../../user/project/deploy_boards.md). **(PREMIUM)** + - Deploy your application to a production environment in a Kubernetes cluster with [Auto Deploy](../../topics/autodevops/index.md#auto-deploy). With GitLab CI/CD you can also: diff --git a/doc/ci/jenkins/index.md b/doc/ci/jenkins/index.md new file mode 100644 index 00000000000..f8a3fab88e3 --- /dev/null +++ b/doc/ci/jenkins/index.md @@ -0,0 +1,232 @@ +--- +comments: false +type: index, howto +--- + +# Migrating from Jenkins + +A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this +easier if you're just getting started, we've collected several resources here that you might find useful +before diving in. + +First of all, our [Quick Start Guide](../quick_start/README.md) contains a good overview of how GitLab CI/CD works. +You may also be interested in [Auto DevOps](../../topics/autodevops/index.md) which can potentially be used to build, test, +and deploy your applications with little to no configuration needed at all. + +Otherwise, read on for important information that will help you get the ball rolling. Welcome +to GitLab! + +## Important differences + +There are some high level differences between the products worth mentioning: + +- With GitLab you don't need a root `pipeline` keyword to wrap everything. +- All jobs within a single stage always run in parallel, and all stages run in sequence. We are planning + to allow certain jobs to break this sequencing as needed with our [directed acyclic graph](https://gitlab.com/gitlab-org/gitlab-ce/issues/47063) + feature. +- The `.gitlab-ci.yml` file is checked in to the root of your repository, much like a Jenkinsfile, but + is in the YAML format (see [complete reference](../yaml/README.md)) instead of a Groovy DSL. It's most + analagous to the declarative Jenkinsfile format. +- GitLab comes with a [container registry](../../user/project/container_registry.md), and we recommend using + container images to set up your build environment. + +## Groovy vs. YAML + +Jenkins Pipelines are based on [Groovy](https://groovy-lang.org/), so the pipeline specification is written as code. +GitLab works a bit differently, we use the more highly structured [YAML](https://yaml.org/) format, which +places scripting elements inside of `script:` blocks separate from the pipeline specification itself. + +This is a strength of GitLab, in that it helps keep the learning curve much simpler to get up and running +and avoids some of the problem of unconstrained complexity which can make your Jenkinsfiles hard to understand +and manage. + +That said, we do of course still value DRY (don't repeat yourself) principles and want to ensure that +behaviors of your jobs can be codified once and applied as needed. You can use the `extends:` syntax to +[templatize your jobs](../yaml/README.md#extends), and `include:` can be used to [bring in entire sets of behaviors](../yaml/README.md#include) +to pipelines in different projects. + +```yaml +.in-docker: + tags: + - docker + image: alpine + +rspec: + extends: + - .in-docker + script: + - rake rspec +``` + +## Artifact publishing + +Artifacts may work a bit differently than you've used them with Jenkins. In GitLab, any job can define +a set of artifacts to be saved by using the `artifacts:` keyword. This can be configured to point to a file +or set of files that can then be persisted from job to job. Read more on our detailed [artifacts documentation](../../user/project/pipelines/job_artifacts.html) + +```yaml +pdf: + script: xelatex mycv.tex + artifacts: + paths: + - ./mycv.pdf + - ./output/ + expire_in: 1 week +``` + +Additionally, we have package management features like a built-in container, NPM, and Maven registry that you +can leverage. You can see the complete list of packaging features (which includes links to documentation) +in the [Packaging section of our documentation](../../README.md#package). + +## Integrated features + +Where you may have used plugins to get things like code quality, unit tests, security scanning, and so on working in Jenkins, +GitLab takes advantage of our connected ecosystem to automatically pull these kinds of results into +your Merge Requests, pipeline details pages, and other locations. You may find that you actually don't +need to configure anything to have these appear. + +If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../README.md#feature-set) has the full list +of bundled features and links to the documentation for each. + +## Converting Declarative Jenkinsfiles + +Declarative Jenkinsfiles contain "Sections" and "Directives" which are used to control the behavior of your +pipelines. There are equivalents for all of these in GitLab, which we've documented below. + +This section is based on the [Jenkinsfile syntax documentation](https://jenkins.io/doc/book/pipeline/syntax/) +and is meant to be a mapping of concepts there to concepts in GitLab. + +### Sections + +#### `agent` + +The agent section is used to define how a pipeline will be executed. For GitLab, we use the [GitLab Runner](../runners/README.md) +to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage +of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users.) The link above will bring you to the documenation which will describe how to get +up and running quickly. We also support using [tags](../runners/README.md#using-tags) to direct different jobs +to different Runners (execution agents). + +The `agent` section also allows you to define which Docker images should be used for execution, for which we use +the [`image`](../yaml/README.md#image) keyword. The `image` can be set on a single job or at the top level, in which +case it will apply to all jobs in the pipeline. + +```yaml +my_job: + image: alpine + ... +``` + +#### `post` + +The `post` section defines the actions that should be performed at the end of the pipeline. GitLab also supports +this through the use of stages. You can define your stages as follows, and any jobs assigned to the `before_pipeline` +or `after_pipeline` stages will run as expected. You can call these stages anything you like. + +```yaml +stages: + - before_pipeline + - build + - test + - deploy + - after_pipeline +``` + +Setting a step to be performed before and after any job can be done via the [`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script). + +```yaml +default: + before_script: + - echo "I run before any jobs starts in the entire pipeline, and can be responsible for setting up the environment." +``` + +#### `stages` + +GitLab CI also lets you define stages, but is a little bit more free-form to configure. The GitLab [`stages` keyword](../yaml/README.md#stages) +is a top level setting that enumerates the list of stages, but you are not required to nest individual jobs underneath +the `stages` section. Any job defined in the `.gitlab-ci.yml` can be made a part of any stage through use of the +[`stage:` keyword](../yaml/README.md#stage). + +Note that, unless otherwise specified, every pipeline is instantiated with a `build`, `test`, and `deploy` stage +which are run in that order. Jobs that have no `stage` defined are placed by default in the `test` stage. +Of course, each job that refers to a stage must refer to a stage that exists in the pipeline configuration. + +```yaml +stages: + - build + - test + - deploy + +my_job: + stage: build + ... +``` + +#### `steps` + +The `steps` section is equivalent to the [`script` section](../yaml/README.md#script) of an individual job. This is +a simple YAML array with each line representing an individual command to be run. + +```yaml +my_job: + script: + - echo "hello! the current time is:" + - time + ... +``` + +### Directives + +#### `environment` + +In GitLab, we use the [`variables` keyword](../yaml/README.md#variables) to define different variables at runtime. +These can also be set up through the GitLab UI, under CI/CD settings. See also our [general documentation on variables](../variables/README.md), +including the section on [protected variables](../variables/README.md#protected-environment-variables) which can be used +to limit access to certain variables to certain environments or runners. + +```yaml +variables: + POSTGRES_USER: user + POSTGRES_PASSWORD: testing_password +``` + +#### `options` + +Here, options for different things exist associated with the object in question itself. For example, options related +to jobs are defined in relation to the job itself. If you're looking for a certain option, you should be able to find +where it's located by searching our [complete configuration reference](../yaml/README.md) page. + +#### `parameters` + +GitLab does not require you to define which variables you want to be available when starting a manual job. A user +can provide any variables they like. + +#### `triggers` / `cron` + +Because GitLab is integrated tightly with git, SCM polling options for triggers are not needed. We support an easy to use +[syntax for scheduling pipelines](../../user/project/pipelines/schedules.md). + +#### `tools` + +GitLab does not support a separate `tools` directive. Our best-practice reccomendation is to use pre-built +container images, which can be cached, and can be built to already contain the tools you need for your pipelines. Pipelines can +be set up to automatically build these images as needed and deploy them to the [container registry](../../user/project/container_registry.md). + +If you're not using container images with Docker/Kubernetes, for example on Mac or FreeBSD, then the `shell` executor does require you to +set up your environment either in advance or as part of the jobs. You could create a `before_script` +action that handles this for you. + +#### `input` + +Similar to the `parameters` keyword, this is not needed because a manual job can always be provided runtime +variable entry. + +#### `when` + +GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be +run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in +our very powerful [`only/except` rules system](../yaml/README.md#onlyexcept-basic) (see also our [advanced syntax](../yaml/README.md#onlyexcept-basic)) + +```yaml +my_job: + only: branches +```
\ No newline at end of file diff --git a/doc/ci/merge_request_pipelines/img/merge_request.png b/doc/ci/merge_request_pipelines/img/merge_request.png Binary files differindex d03fdc6a885..bb64e17cc91 100644 --- a/doc/ci/merge_request_pipelines/img/merge_request.png +++ b/doc/ci/merge_request_pipelines/img/merge_request.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png Binary files differindex 58d5581f628..6d4b66824e1 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png Binary files differindex 0a84e61d284..3ee9d8ec93b 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/img/merge_request_pipeline_config.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index a13857bee25..126e12e460f 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -6,7 +6,6 @@ last_update: 2019-07-03 # Pipelines for Merged Results **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. -> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222), but [can be enabled manually](#enabling-pipelines-for-merged-results). It's possible for your source and target branches to diverge, which can result in the scenario that source branch's pipeline was green, the target's pipeline was green, @@ -25,8 +24,10 @@ new ref and a pipeline for this ref validates the result prior to merging. There are some cases where creating a combined ref is not possible or not wanted. For example, a source branch that has conflicts with the target branch -or a merge request that is still in WIP status. In this case, the merge request pipeline falls back to a "detached" state -and runs on the source branch ref as if it was a regular pipeline. +or a merge request that is still in WIP status. In this case, +GitLab doesn't create a merge commit and the pipeline runs on source branch, instead, +which is a default behavior of [Pipelines for merge requests](../index.md) + i.e. `detached` label is shown to the pipelines. The detached state serves to warn you that you are working in a situation subjected to merge problems, and helps to highlight that you should @@ -34,15 +35,14 @@ get out of WIP status or resolve merge conflicts as soon as possible. ## Requirements and limitations -Pipelines for merged results require: +Pipelines for merged results require a [GitLab Runner][runner] 11.9 or newer. -- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. -- [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. +[runner]: https://gitlab.com/gitlab-org/gitlab-runner In addition, pipelines for merged results have the following limitations: - Forking/cross-repo workflows are not currently supported. To follow progress, - see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). + see [#11934](https://gitlab.com/gitlab-org/gitlab-ee/issues/11934). - This feature is not available for [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). @@ -61,18 +61,32 @@ CAUTION: **Warning:** Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](../index.md#configuring-pipelines-for-merge-requests), otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. -## Merge Trains **(PREMIUM)** +## Troubleshooting -Read the [documentation on Merge Trains](merge_trains/index.md). +### Pipelines for merged results not created even with new change pushed to merge request -<!-- ## Troubleshooting +Can be caused by some disabled feature flags. Please make sure that +the following feature flags are enabled on your GitLab instance: -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +- `:ci_use_merge_request_ref` +- `:merge_ref_auto_sync` -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +To check these feature flag values, please ask administrator to execute the following commands: + +```shell +> sudo gitlab-rails console # Login to Rails console of GitLab instance. +> Feature.enabled?(:ci_use_merge_request_ref) # Check if it's enabled or not. +> Feature.enable(:ci_use_merge_request_ref) # Enable the feature flag. +``` + +## Using Merge Trains **(PREMIUM)** + +By enabling [Pipelines for merged results](#pipelines-for-merged-results-premium), +GitLab will [automatically display](merge_trains/index.md#how-to-add-a-merge-request-to-a-merge-train) +a **Start/Add Merge Train button** as the most recommended merge strategy. + +Generally, this is a safer option than merging merge requests immediately as your +merge request will be evaluated with an expected post-merge result before the actual +merge happens. + +For more information, read the [documentation on Merge Trains](merge_trains/index.md). diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png Binary files differindex 1561fdcc7a5..d7720ac1143 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_cancel_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png Binary files differindex fb0af43556e..9da959ad440 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_config_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png Binary files differnew file mode 100644 index 00000000000..8a795fff432 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_failure.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png Binary files differnew file mode 100644 index 00000000000..03bc61129ba --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_immediate_merge.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png Binary files differnew file mode 100644 index 00000000000..ec4b157d428 --- /dev/null +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_position_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png Binary files differindex f20108157d2..a4d0c8cf0e6 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png Binary files differindex 62c2f2f5ff5..45762b8e85e 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/img/merge_train_start_when_pipeline_succeeds_v12_0.png diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index 57358434c02..80a1c264bc4 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -6,7 +6,6 @@ last_update: 2019-07-03 # Merge Trains **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. -> This feature is disabled by default, but [can be enabled manually](#enabling-merge-trains). [Pipelines for merged results](../index.md#pipelines-for-merged-results-premium) introduces running a build on the result of the merged code prior to merging, as a way to keep master green. @@ -33,35 +32,40 @@ Merge trains have the following requirements and limitations: - This feature requires that [pipelines for merged results](../index.md#pipelines-for-merged-results-premium) are **configured properly**. -- Each merge train can generate a merge ref and run a pipeline **one at a time**. - We plan to make the pipelines for merged results - [run in parallel](https://gitlab.com/gitlab-org/gitlab-ee/issues/11222) in a - future release. +- Each merge train can run a maximum of **four** pipelines in parallel. + If more than four merge requests are added to the merge train, the merge requests + will be queued until a slot in the merge train is free. There is no limit to the + number of merge requests that can be queued. +- This feature does not support [squash and merge](../../../../user/project/merge_requests/squash_and_merge.md). -## Enabling Merge Trains - -To enable merge trains at the project level: - -1. Visit your project's **Settings > General** and expand **Merge requests**. -1. Check **Allow merge trains**. -1. Click **Save changes** button. - - +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +Watch this video for a demonstration on [how parallel execution +of Merge Trains can prevent commits from breaking the default +branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). ## How to add a merge request to a merge train To add a merge request to a merge train: -1. Click "Start/Add merge train" button in a merge request +1. Visit a merge request. +1. Click the **Start/Add to merge train** button.  ## How to remove a merge request from a merge train -1. Click "Remove from merge train" button in the merge request widget. +1. Visit a merge request. +1. Click the **Remove from merge train** button.  +## How to view a merge request's current position on the merge train + +After a merge request has been added to the merge train, the merge request's +current position will be displayed under the pipeline widget: + + + ## Start/Add to merge train when pipeline succeeds You can add a merge request to a merge train only when the latest pipeline in the @@ -70,19 +74,71 @@ the merge request to a train because the current change of the merge request may be broken thus it could affect the following merge requests. In this case, you can schedule to add the merge request to a merge train **when the latest -pipeline succeeds**. You can see the following button instead of the regular "Start/Add merge train" +pipeline succeeds** (This pipeline is [Pipelines for merged results](../index.md), not Pipelines for merge train). +You can see the following button instead of the regular **Start/Add to merge train** button while the latest pipeline is running.  -<!-- ## Troubleshooting +## Immediately merge a merge request with a merge train + +In case, you have a high-priority merge request (e.g. critical patch) to be merged urgently, +you can use **Merge Immediately** option for bypassing the merge train. +This is the fastest option to get the change merged into the target branch. + + + +However, every time you merge a merge request immediately, it could affect the +existing merge train to be reconstructed, specifically, it regenerates expected +merge commits and pipelines. This means, merging immediately essentially wastes +CI resources. + +## Troubleshooting + +### Merge request dropped from the merge train immediately + +If a merge request is not mergeable (for example, it's WIP, there is a merge +conflict, etc), your merge request will be dropped from the merge train automatically. + +In these cases, the reason for dropping the merge request is in the **system notes**. + +To check the reason: + +1. Open the merge request that was dropped from the merge train. +1. Open the **Discussion** tab. +1. Find a system note that includes either: + - The text **... removed this merge request from the merge train because ...** + - **... aborted this merge request from the merge train because ...** + The reason is given in the text after the **because ...** phrase. + + + +### Merge When Pipeline Succeeds cannot be chosen + +[Merge When Pipeline Succeeds](../../../../user/project/merge_requests/merge_when_pipeline_succeeds.md) +is unavailable when +[Pipelines for Merged Results is enabled](../index.md#enabling-pipelines-for-merged-results). + +Follow [this issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/12267) to +track progress on this issue. + +### Merge Train disturbs your workflow + +First of all, please check if [merge immediately](#immediately-merge-a-merge-request-with-a-merge-train) +is available as a workaround in your workflow. This is the most recommended +workaround you'd be able to take immediately. If it's not available or acceptable, +please read through this section. + +Merge train is enabled by default when you enable [Pipelines for merged results](../index.md), +however, you can forcibly disable this feature by disabling the feature flag `:merge_trains_enabled`. +After you disabled this feature, all the existing merge trains will be aborted and +you will no longer see the **Start/Add Merge Train** button in merge requests. -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +To check if the feature flag is enabled on your GitLab instance, +please ask administrator to execute the following commands: -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +```shell +> sudo gitlab-rails console # Login to Rails console of GitLab instance. +> Feature.enabled?(:merge_trains_enabled) # Check if it's enabled or not. +> Feature.disable(:merge_trains_enabled) # Disable the feature flag. +```
\ No newline at end of file diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index 463b9194c58..61f260cb70d 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -171,6 +171,26 @@ In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the upstream pipeline will be passed to the `downstream-job` job, and will be available within the context of all downstream builds. +NOTE: **Tip:** +Upstream pipelines take precedence over downstream ones. If there are two +variables with the same name defined in both upstream and downstream projects, +the ones defined in the upstream project will take precedence. + +### Mirroring status from upstream pipeline + +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from master is +replicated to the bridge job. + +Example: + +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` + ### Limitations Because bridge jobs are a little different to regular jobs, it is not @@ -185,5 +205,5 @@ Some features are not implemented yet. For example, support for environments. - `stage` - `allow_failure` - `only` and `except` -- `when` +- `when` (only with `on_success`, `on_failure`, and `always` values) - `extends` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 06a81c3d0c7..eaa6efc526d 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -6,6 +6,11 @@ type: reference > Introduced in GitLab 8.8. +NOTE: **Tip:** +Watch our +["Mastering continuous software development"](https://about.gitlab.com/webcast/mastering-ci-cd/) +webcast to see a comprehensive demo of GitLab CI/CD pipeline. + ## Introduction Pipelines are the top-level component of continuous integration, delivery, and deployment. @@ -318,6 +323,20 @@ stage has a job with a manual action.  +### Specifying variables when running manual jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30485) in GitLab 12.2. + +When running manual jobs you can supply additional job specific variables. + +You can do this from the job page of the manual job you want to run with +additional variables. + +This is useful when you want to alter the execution of a job by using +environment variables. + + + ### Delay a job in a pipeline graph > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. @@ -345,7 +364,7 @@ GitLab provides API endpoints to: - [Triggering pipelines through the API](triggers/README.md). - [Pipeline triggers API](../api/pipeline_triggers.md). -### Start multiple manual actions in a stage +### Start multiple manual actions in a stage > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27188) in GitLab 11.11. @@ -358,6 +377,10 @@ This functionality is only available: - For users with at least Developer access. - If the the stage contains [manual actions](#manual-actions-from-pipeline-graphs). +## Most Recent Pipeline + +There's a link to the latest pipeline for the last commit of a given branch at `/project/pipelines/[branch]/latest`. Also, `/project/pipelines/latest` will redirect you to the latest pipeline for the last commit on the project's default branch. + ## Security on protected branches A strict security model is enforced when pipelines are executed on diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 0480b83d183..5626428bfc3 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -13,6 +13,10 @@ NOTE: **Note:** Please keep in mind that only project Maintainers and Admin users have the permissions to access a project's settings. +NOTE: **Note:** +Coming over to GitLab from Jenkins? Check out our [reference](../jenkins/index.md) +for converting your pre-existing pipelines over to our format. + GitLab offers a [continuous integration][ci] service. If you [add a `.gitlab-ci.yml` file][yaml] to the root directory of your repository, and configure your GitLab project to use a [Runner], then each commit or diff --git a/doc/ci/quick_start/img/build_log.png b/doc/ci/quick_start/img/build_log.png Binary files differindex 2bf0992c50e..16698629edc 100644 --- a/doc/ci/quick_start/img/build_log.png +++ b/doc/ci/quick_start/img/build_log.png diff --git a/doc/ci/quick_start/img/builds_status.png b/doc/ci/quick_start/img/builds_status.png Binary files differindex 58978e23978..b4aeeb988d2 100644 --- a/doc/ci/quick_start/img/builds_status.png +++ b/doc/ci/quick_start/img/builds_status.png diff --git a/doc/ci/quick_start/img/pipelines_status.png b/doc/ci/quick_start/img/pipelines_status.png Binary files differindex 06d1559f5d2..39a77a26b25 100644 --- a/doc/ci/quick_start/img/pipelines_status.png +++ b/doc/ci/quick_start/img/pipelines_status.png diff --git a/doc/ci/quick_start/img/runners_activated.png b/doc/ci/quick_start/img/runners_activated.png Binary files differindex cd83c1a7e4c..ac09e1d0137 100644 --- a/doc/ci/quick_start/img/runners_activated.png +++ b/doc/ci/quick_start/img/runners_activated.png diff --git a/doc/ci/review_apps/img/review_button.png b/doc/ci/review_apps/img/review_button.png Binary files differindex 0b231c50858..4f1cf4d7cfd 100644 --- a/doc/ci/review_apps/img/review_button.png +++ b/doc/ci/review_apps/img/review_button.png diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 9b89988bf42..8ab7982fd65 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -147,11 +147,11 @@ Once you have the route mapping set up, it will take effect in the following loc - In the diff for a merge request, comparison, or commit. -  +  - In the blob file view. -  +  ## Visual Reviews **(STARTER)** diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 03a219e03ca..269bd5c3428 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -62,7 +62,7 @@ You can only register a shared Runner if you are an admin of the GitLab instance 1. Grab the shared-Runner token on the `admin/runners` page -  +  1. [Register the Runner][register] @@ -88,7 +88,7 @@ visit the project you want to make the Runner work for in GitLab: ## Registering a group Runner -Creating a group Runner requires Maintainer permissions for the group. To create a +Creating a group Runner requires Owner permissions for the group. To create a group Runner visit the group you want to make the Runner work for in GitLab: 1. Go to **Settings > CI/CD** to obtain the token @@ -124,9 +124,9 @@ To lock/unlock a Runner: ## Assigning a Runner to another project -If you are Maintainer on a project where a specific Runner is assigned to, and the +If you are an Owner on a project where a specific Runner is assigned to, and the Runner is not [locked only to that project](#locking-a-specific-runner-from-being-enabled-for-other-projects), -you can enable the Runner also on any other project where you have Maintainer permissions. +you can enable the Runner also on any other project where you have Owner permissions. To enable/disable a Runner in your project: @@ -250,7 +250,7 @@ When you [register a Runner][register], its default behavior is to **only pick** [tagged jobs](../yaml/README.md#tags). NOTE: **Note:** -Maintainer [permissions](../../user/permissions.md) are required to change the +Owner [permissions](../../user/permissions.md) are required to change the Runner settings. To make a Runner pick untagged jobs: @@ -319,21 +319,21 @@ How this feature will work: 1. You set the _maximum job timeout_ for a Runner to 24 hours 1. You set the _CI/CD Timeout_ for a project to **2 hours** 1. You start a job -1. The job, if running longer, will be timeouted after **2 hours** +1. The job, if running longer, will be timed out after **2 hours** **Example 2 - Runner timeout not configured** 1. You remove the _maximum job timeout_ configuration from a Runner 1. You set the _CI/CD Timeout_ for a project to **2 hours** 1. You start a job -1. The job, if running longer, will be timeouted after **2 hours** +1. The job, if running longer, will be timed out after **2 hours** **Example 3 - Runner timeout smaller than project timeout** 1. You set the _maximum job timeout_ for a Runner to **30 minutes** 1. You set the _CI/CD Timeout_ for a project to 2 hours 1. You start a job -1. The job, if running longer, will be timeouted after **30 minutes** +1. The job, if running longer, will be timed out after **30 minutes** ### Be careful with sensitive information @@ -373,12 +373,12 @@ attacker. To reset the token: -1. Go to **Settings > CI/CD** for a specified Project -1. Expand the **General pipelines settings** section -1. Find the **Runner token** form field and click the **Reveal value** button -1. Delete the value and save the form +1. Go to **Settings > CI/CD** for a specified Project. +1. Expand the **General pipelines settings** section. +1. Find the **Runner token** form field and click the **Reveal value** button. +1. Delete the value and save the form. 1. After the page is refreshed, expand the **Runners settings** section - and check the registration token - it should be changed + and check the registration token - it should be changed. From now on the old token is not valid anymore and will not allow to register a new Runner to the project. If you are using any tools to provision and diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 697452cee83..9ea113969c8 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -25,6 +25,12 @@ variables: MYSQL_ROOT_PASSWORD: "<your_mysql_password>" ``` +NOTE: **Note:** +The `MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables can't be set in the GitLab UI. +To set them, assign them to a variable [in the UI](../variables/README.md#via-the-ui), +and then assign that variable to the +`MYSQL_DATABASE` and `MYSQL_ROOT_PASSWORD` variables in your `.gitlab-ci.yml`. + And then configure your application to use the database, for example: ```yaml diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 960346ac11a..142f4f262e5 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -25,6 +25,13 @@ variables: POSTGRES_PASSWORD: "" ``` +NOTE: **Note:** +The `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD` variables can't be set in +the GitLab UI. To set them, assign them to a variable +[in the UI](../variables/README.md#via-the-ui), and then assign that +variable to the `POSTGRES_DB`, `POSTGRES_USER`, and `POSTGRES_PASSWORD` variables in +your `.gitlab-ci.yml`. + And then configure your application to use the database, for example: ```yaml @@ -70,7 +77,7 @@ template1=# CREATE USER runner WITH PASSWORD '$password' CREATEDB; ``` *__Note:__ Notice that we created the user with the privilege to be able to -create databases (`CREATEDB`). In the following steps we will create a database +create databases (`CREATEDB`). In the following steps we will create a database explicitly for that user but having that privilege can be useful if in your testing framework you have tools that drop and create databases.* diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index 69591ed605c..b6aebd3bd78 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -57,44 +57,44 @@ to access it. This is where an SSH key pair comes in handy. 1. Modify your `.gitlab-ci.yml` with a `before_script` action. In the following example, a Debian based image is assumed. Edit to your needs: - ```yaml - before_script: - ## - ## Install ssh-agent if not already installed, it is required by Docker. - ## (change apt-get to yum if you use an RPM-based image) - ## - - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' - - ## - ## Run ssh-agent (inside the build environment) - ## - - eval $(ssh-agent -s) - - ## - ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store - ## We're using tr to fix line endings which makes ed25519 keys work - ## without extra base64 encoding. - ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556 - ## - - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null - - ## - ## Create the SSH directory and give it the right permissions - ## - - mkdir -p ~/.ssh - - chmod 700 ~/.ssh - - ## - ## Optionally, if you will be using any Git commands, set the user name and - ## and email. - ## - #- git config --global user.email "user@example.com" - #- git config --global user.name "User name" - ``` - - NOTE: **Note:** - The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally - or per-job. + ```yaml + before_script: + ## + ## Install ssh-agent if not already installed, it is required by Docker. + ## (change apt-get to yum if you use an RPM-based image) + ## + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + + ## + ## Run ssh-agent (inside the build environment) + ## + - eval $(ssh-agent -s) + + ## + ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store + ## We're using tr to fix line endings which makes ed25519 keys work + ## without extra base64 encoding. + ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556 + ## + - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - + + ## + ## Create the SSH directory and give it the right permissions + ## + - mkdir -p ~/.ssh + - chmod 700 ~/.ssh + + ## + ## Optionally, if you will be using any Git commands, set the user name and + ## and email. + ## + #- git config --global user.email "user@example.com" + #- git config --global user.name "User name" + ``` + + NOTE: **Note:** + The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally + or per-job. 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys). @@ -118,9 +118,9 @@ on, and use that key for all projects that are run on this machine. 1. Then from the terminal login as the `gitlab-runner` user: - ``` - sudo su - gitlab-runner - ``` + ``` + sudo su - gitlab-runner + ``` 1. Generate the SSH key pair as described in the instructions to [generate an SSH key](../../ssh/README.md#generating-a-new-ssh-key-pair). diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index d1f9aa03b6b..2a382f18038 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -97,17 +97,6 @@ overview of the time the triggers were last used.  -## Taking ownership of a trigger - -> **Note**: -GitLab 9.0 introduced a trigger ownership to solve permission problems. - -Each created trigger when run will impersonate their associated user including -their access to projects and their project permissions. - -You can take ownership of existing triggers by clicking *Take ownership*. -From now on the trigger will be run as you. - ## Revoking a trigger You can revoke a trigger any time by going at your project's @@ -282,8 +271,7 @@ Old triggers, created before GitLab 9.0 will be marked as legacy. Triggers with the legacy label do not have an associated user and only have access to the current project. They are considered deprecated and will be -removed with one of the future versions of GitLab. You are advised to -[take ownership](#taking-ownership-of-a-trigger) of any legacy triggers. +removed with one of the future versions of GitLab. [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 [ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 42dd4f08ed8..5a15b907da0 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -94,7 +94,10 @@ This means that the value of the variable will be hidden in job logs, though it must match certain requirements to do so: - The value must be in a single line. -- The value must only consist of characters from the Base64 alphabet, defined in [RFC4648](https://tools.ietf.org/html/rfc4648). +- The value must only consist of characters from the Base64 alphabet (RFC4648). + + [In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-ce/issues/63043) + and newer, `@` and `:` are also valid values. - The value must be at least 8 characters long. - The value must not use variables. @@ -273,6 +276,7 @@ export CI_RUNNER_ID="10" export CI_RUNNER_DESCRIPTION="my runner" export CI_RUNNER_TAGS="docker, linux" export CI_SERVER="yes" +export CI_SERVER_HOST="example.com" export CI_SERVER_NAME="GitLab" export CI_SERVER_REVISION="70606bf" export CI_SERVER_VERSION="8.9.0" @@ -369,8 +373,11 @@ variables take precedence over those defined in `.gitlab-ci.yml`. ## Unsupported variables There are cases where some variables cannot be used in the context of a -`.gitlab-ci.yml` definition (for example under `script`). Read more -about which variables are [not supported](where_variables_can_be_used.md). +`.gitlab-ci.yml` definition (for example under `script`). Read more about which variables are [not supported](where_variables_can_be_used.md). + +## Where variables can be used + +Click [here](where_variables_can_be_used.md) for a section that describes where and how the different types of variables can be used. ## Advanced use @@ -389,7 +396,7 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. -### Limiting environment scopes of environment variables **(PREMIUM)** +### Limiting environment scopes of environment variables You can limit the environment scope of a variable by [defining which environments][envs] it can be available for. @@ -480,81 +487,86 @@ Below you can find supported syntax reference: 1. Equality matching using a string - > Example: `$VARIABLE == "some value"` + Examples: - > Example: `$VARIABLE != "some value"` (introduced in GitLab 11.11) + - `$VARIABLE == "some value"` + - `$VARIABLE != "some value"` (introduced in GitLab 11.11) - You can use equality operator `==` or `!=` to compare a variable content to a - string. We support both, double quotes and single quotes to define a string - value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` - are supported. `"some value" == $VARIABLE` is correct too. + You can use equality operator `==` or `!=` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. 1. Checking for an undefined value - > Example: `$VARIABLE == null` + Examples: - > Example: `$VARIABLE != null` (introduced in GitLab 11.11) + - `$VARIABLE == null` + - `$VARIABLE != null` (introduced in GitLab 11.11) - It sometimes happens that you want to check whether a variable is defined - or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression is going to evaluate to truth if - variable is not defined when `==` is used, or to falsey if `!=` is used. + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined when `==` is used, or to falsey if `!=` is used. 1. Checking for an empty variable - > Example: `$VARIABLE == ""` + Examples: - > Example: `$VARIABLE != ""` (introduced in GitLab 11.11) + - `$VARIABLE == ""` + - `$VARIABLE != ""` (introduced in GitLab 11.11) - If you want to check whether a variable is defined, but is empty, you can - simply compare it against an empty string, like `$VAR == ''` or non-empty - string `$VARIABLE != ""`. + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''` or non-empty + string `$VARIABLE != ""`. 1. Comparing two variables - > Example: `$VARIABLE_1 == $VARIABLE_2` + Examples: - > Example: `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) + - `$VARIABLE_1 == $VARIABLE_2` + - `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) - It is possible to compare two variables. This is going to compare values - of these variables. + It is possible to compare two variables. This is going to compare values + of these variables. 1. Variable presence check - > Example: `$STAGING` + Example: `$STAGING` - If you only want to create a job when there is some variable present, - which means that it is defined and non-empty, you can simply use - variable name as an expression, like `$STAGING`. If `$STAGING` variable - is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to a string, with length higher than zero. - Variable that contains only whitespace characters is not an empty variable. + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. 1. Pattern matching (introduced in GitLab 11.0) - > Example: `$VARIABLE =~ /^content.*/` + Examples: - > Example: `$VARIABLE_1 !~ /^content.*/` (introduced in GitLab 11.11) + - `$VARIABLE =~ /^content.*/` + - `$VARIABLE_1 !~ /^content.*/` (introduced in GitLab 11.11) - It is possible perform pattern matching against a variable and regular - expression. Expression like this evaluates to truth if matches are found - when using `=~`. It evaluates to truth if matches are not found when `!~` is used. + It is possible perform pattern matching against a variable and regular + expression. Expression like this evaluates to truth if matches are found + when using `=~`. It evaluates to truth if matches are not found when `!~` is used. - Pattern matching is case-sensitive by default. Use `i` flag modifier, like - `/pattern/i` to make a pattern case-insensitive. + Pattern matching is case-sensitive by default. Use `i` flag modifier, like + `/pattern/i` to make a pattern case-insensitive. 1. Conjunction / Disjunction ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27925) in GitLab 12.0) - > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` - - > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + Examples: - > Example: `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` + - `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + - `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` - It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise - supported syntax may be used in a conjunctive or disjunctive statement. - Precedence of operators follows standard Ruby 2.5 operation - [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). + It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise + supported syntax may be used in a conjunctive or disjunctive statement. + Precedence of operators follows standard Ruby 2.5 operation + [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). ## Debug tracing @@ -644,6 +656,8 @@ Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-mach ++ CI_PROJECT_DIR=/builds/gitlab-examples/ci-debug-trace ++ export CI_SERVER=yes ++ CI_SERVER=yes +++ export 'CI_SERVER_HOST=example.com' +++ CI_SERVER_HOST='example.com' ++ export 'CI_SERVER_NAME=GitLab CI' ++ CI_SERVER_NAME='GitLab CI' ++ export CI_SERVER_VERSION= @@ -678,6 +692,8 @@ Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-mach ++ CI_JOB_NAME=debug_trace ++ export CI_JOB_STAGE=test ++ CI_JOB_STAGE=test +++ export CI_SERVER_HOST=example.com +++ CI_SERVER_HOST=example.com ++ export CI_SERVER_NAME=GitLab ++ CI_SERVER_NAME=GitLab ++ export CI_SERVER_VERSION=8.14.3-ee diff --git a/doc/ci/variables/img/custom_variables_output.png b/doc/ci/variables/img/custom_variables_output.png Binary files differindex 29f5c63b3d9..797e9ec07b9 100644 --- a/doc/ci/variables/img/custom_variables_output.png +++ b/doc/ci/variables/img/custom_variables_output.png diff --git a/doc/ci/variables/img/new_custom_variables_example.png b/doc/ci/variables/img/new_custom_variables_example.png Binary files differindex efe104efe4c..bb60e6bab21 100644 --- a/doc/ci/variables/img/new_custom_variables_example.png +++ b/doc/ci/variables/img/new_custom_variables_example.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differindex 3c8c59720cf..c77c5cb7764 100644 --- a/doc/ci/variables/img/override_variable_manual_pipeline.png +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/img/variable_types_usage_example.png b/doc/ci/variables/img/variable_types_usage_example.png Binary files differindex 0e8bde891fe..c2ae32fd048 100644 --- a/doc/ci/variables/img/variable_types_usage_example.png +++ b/doc/ci/variables/img/variable_types_usage_example.png diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index e911e97d3c8..409f7d62538 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -30,7 +30,7 @@ future GitLab releases.** | `CI_BUILDS_DIR` | all | 11.10 | Top-level directory where builds are executed. | | `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | | `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | -| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. Only populated when there is a merge request associated with the pipeline. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a merge request. Only populated when there is a merge request associated with the pipeline. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | | `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | | `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built | @@ -101,6 +101,7 @@ future GitLab releases.** | `CI_RUNNER_TAGS` | 8.10 | 0.5 | The defined runner tags | | `CI_RUNNER_VERSION` | all | 10.6 | GitLab Runner version that is executing the current job | | `CI_SERVER` | all | all | Mark that job is executed in CI environment | +| `CI_SERVER_HOST` | 12.1 | all | Host component of the GitLab instance URL, without protocol and port (like gitlab.example.com) | | `CI_SERVER_NAME` | all | all | The name of CI server that is used to coordinate jobs | | `CI_SERVER_REVISION` | all | all | GitLab revision that is used to schedule jobs | | `CI_SERVER_VERSION` | all | all | GitLab version that is used to schedule jobs | diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 474db05de06..1368764bcf8 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -100,6 +100,7 @@ The following table lists available parameters for jobs: | [`stage`](#stage) | Defines a job stage (default: `test`). | | [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | | [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | +| [`rules`](#rules) | List of coniditions to evaluate and determine selected attributes of a build and whether or not it is created. May not be used alongside `only`/`except`. | [`tags`](#tags) | List of tags which are used to select Runner. | | [`allow_failure`](#allow_failure) | Allow job to fail. Failed job doesn't contribute to commit status. | | [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | @@ -119,6 +120,35 @@ The following table lists available parameters for jobs: NOTE: **Note:** Parameters `types` and `type` are [deprecated](#deprecated-parameters). +## Setting default parameters + +Some parameters can be set globally as the default for all jobs using the +`default:` keyword. Default parameters can then be overridden by job-specific +configuration. + +The following job parameters can be defined inside a `default:` block: + +- [`image`](#image) +- [`services`](#services) +- [`before_script`](#before_script-and-after_script) +- [`after_script`](#before_script-and-after_script) +- [`cache`](#cache) + +In the following example, the `ruby:2.5` image is set as the default for all +jobs except the `rspec 2.6` job, which uses the `ruby:2.6` image: + +```yaml +default: + image: ruby:2.5 + +rspec: + script: bundle exec rspec + +rspec 2.6: + image: ruby:2.6 + script: bundle exec rspec +``` + ## Parameter details The following are detailed explanations for parameters used to configure CI/CD pipelines. @@ -239,8 +269,9 @@ It's possible to overwrite the globally defined `before_script` and `after_scrip if you set it per-job: ```yaml -before_script: - - global before script +default: + before_script: + - global before script job: before_script: @@ -356,7 +387,7 @@ In addition, `only` and `except` allow the use of special keywords: | `triggers` | For pipelines created using a trigger token. | | `web` | For pipelines created using **Run pipeline** button in GitLab UI (under your project's **Pipelines**). | | `merge_requests` | When a merge request is created or updated (See [pipelines for merge requests](../merge_request_pipelines/index.md)). | -| `chats` | For jobs created using a [GitLab ChatOps](../chatops/README.md) command. | +| `chat` | For jobs created using a [GitLab ChatOps](../chatops/README.md) command. | In the example below, `job` will run only for refs that start with `issue-`, whereas all branches will be skipped: @@ -475,7 +506,7 @@ Feature.enable(:allow_unsafe_ruby_regexp) ### `only`/`except` (advanced) CAUTION: **Warning:** -This an _alpha_ feature, and it is subject to change at any time without +This is an _alpha_ feature, and it is subject to change at any time without prior notice! GitLab supports both simple and complex strategies, so it's possible to use an @@ -488,10 +519,24 @@ Four keys are available: - `changes` - `kubernetes` -If you use multiple keys under `only` or `except`, they act as an AND. The logic is: +If you use multiple keys under `only` or `except`, the keys will be evaluated as a +single conjoined expression. That is: + +- `only:` means "include this job if all of the conditions match". +- `except:` means "exclude this job if any of the conditions match". + +With `only`, individual keys are logically joined by an AND: > (any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active) +`except` is implemented as a negation of this complete expression: + +> NOT((any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active)) + +This, more intuitively, means the keys join by an OR. A functionally equivalent expression: + +> (any of refs) OR (any of variables) OR (any of changes) OR (if kubernetes is active) + #### `only:refs`/`except:refs` > `refs` policy introduced in GitLab 10.0. @@ -646,6 +691,125 @@ In the scenario above, if a merge request is created or updated that changes either files in `service-one` directory or the `Dockerfile`, GitLab creates and triggers the `docker build service one` job. +### `rules` + +Using `rules` allows for a list of individual rule objects to be evaluated +*in order*, until one matches and dynamically provides attributes to the job. + +Available rule clauses include: + +- `if` (similar to [`only:variables`](#onlyvariablesexceptvariables)). +- `changes` (same as [`only:changes`](#onlychangesexceptchanges)). + +For example, using `if`: + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH == "master"' # This rule will be evaluated + when: always + - if: '$VAR =~ /pattern/' # This rule will only be evaluated if the first does not match + when: manual + - when: on_success # A Rule entry with no conditional clauses evaluates to true. If neither of the first two Rules match, this one will and set job:when to "on_success" +``` + +If the first rule does not match, further rules will be evaluated sequentially +until a match is found. The above configuration will specify that `job` should +be built and run for every pipeline on merge requests targeting `master`, +regardless of the status of other builds. + +#### `rules:if` + +`rules:if` differs slightly from `only:variables` by accepting only a single +expression string, rather than an array of them. Any set of expressions to be +evaluated should be conjoined into a single expression using `&&` or `||`. For example: + +```yaml +job: + script: "echo Hello, Rules!" + rules: + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH == "master"' # This rule will be evaluated + when: always + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH =~ /^feature/' # This rule will only be evaluated if the target branch is not "master" + when: manual + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH' # If neither of the first two match but the simple presence does, we set to "on_success" by default +``` + +If none of the provided rules match, the job will be set to `when:never`, and +not included in the pipeline. If `rules:when` is not included in the configuration +at all, the behavior defaults to `job:when`, which continues to default to +`on_success`. + +#### `rules:changes` + +`changes` works exactly the same way as [`only`/`except`](#onlychangesexceptchanges), +accepting an array of paths. The following configuration configures a job to be +run manually if `Dockerfile` has changed OR `$VAR == "string value"`. Otherwise +it is set to `when:on_success` by the last rule, where 0 clauses evaluate as +vacuously true. + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + when: manual + - if: '$VAR == "string value"' + when: manual # Will include the job and set to when:manual if the expression evaluates to true, after the `changes:` rule fails to match. + - when: on_success # If neither of the first rules match, set to on_success + +``` + +#### Complex Rule Clauses + +To conjoin `if` and `changes` clauses with an AND, use them in the same rule. +Here we run the job manually if `Dockerfile` or any file in `docker/scripts/` +has changed AND `$VAR == "string value"`. Otherwise, the job will not be +included in the pipeline. + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$VAR == "string value"' + changes: # Will include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + - docker/scripts/* + when: manual + # - when: never would be redundant here, this is implied any time rules are listed. +``` + +The only clauses currently available are `if` and `changes`. Keywords such as +`branches` or `refs` that are currently available for `only`/`except` are not +yet available in `rules` as they are being individually considered for their +usage and behavior in the newer context. + +#### Permitted attributes + +The only job attributes currently set by `rules` are `when` and `start_in`, if +`when` is set to `delayed`. A job will be included in a pipeline if `when` is +evaluated to any value except `never`. + +Delayed jobs require a `start_in` value, so rule objects do as well. For example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: # Will include the job and delay 3 hours when the Dockerfile has changed + - Dockerfile + when: delayed + start_in: '3 hours' + - when: on_success # Otherwise include the job and set to run normally + +``` + +Additional Job configuration may be added to rules in the future, if something +useful isn't available, please open an issue on +[Gitlab CE](https://www.gitlab.com/gitlab-org/gitlab-ce/issues). + ### `tags` `tags` is used to select specific Runners from the list of all Runners that are @@ -974,7 +1138,7 @@ review_app: stop_review_app: stage: deploy variables: - GIT_STRATEGY: none + GIT_STRATEGY: none script: make delete-app when: manual environment: @@ -1538,7 +1702,7 @@ dashboards. The `license_management` report collects [Licenses](../../user/project/merge_requests/license_management.md) as artifacts. -The collected License Management report will be uploaded to GitLab as an artifact and will +The collected License Compliance report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. @@ -1635,6 +1799,84 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. +### `needs` + +> Introduced in GitLab 12.2. + +The `needs:` keyword enables executing jobs out-of-order, allowing you to implement +a [directed acyclic graph](../directed_acyclic_graph/index.md) in your `.gitlab-ci.yml`. + +This lets you run some jobs without waiting for other ones, disregarding stage ordering +so you can have multiple stages running concurrently. + +Let's consider the following example: + +```yaml +linux:build: + stage: build + +mac:build: + stage: build + +linux:rspec: + stage: test + needs: ["linux:build"] + +linux:rubocop: + stage: test + needs: ["linux:build"] + +mac:rspec: + stage: test + needs: ["mac:build"] + +mac:rubocop: + stage: test + needs: ["mac:build"] + +production: + stage: deploy +``` + +This example creates three paths of execution: + +- Linux path: the `linux:rspec` and `linux:rubocop` jobs will be run as soon + as the `linux:build` job finishes without waiting for `mac:build` to finish. + +- macOS path: the `mac:rspec` and `mac:rubocop` jobs will be run as soon + as the `mac:build` job finishes, without waiting for `linux:build` to finish. + +- The `production` job will be executed as soon as all previous jobs + finish; in this case: `linux:build`, `linux:rspec`, `linux:rubocop`, + `mac:build`, `mac:rspec`, `mac:rubocop`. + +#### Requirements and limitations + +1. If `needs:` is set to point to a job that is not instantiated + because of `only/except` rules or otherwise does not exist, the + job will fail. +1. Note that on day one of the launch, we are temporarily limiting the + maximum number of jobs that a single job can need in the `needs:` array. Track + our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7541) + for details on the current limit. +1. If you use `dependencies:` with `needs:`, it's important that you + do not mark a job as having a dependency on something that won't + have been run at the time it needs it. It's better to use both + keywords in this case so that GitLab handles the ordering appropriately. +1. It is impossible for now to have `needs: []` (empty needs), + the job always needs to depend on something, unless this is the job + in the first stage (see [gitlab-ce#65504](https://gitlab.com/gitlab-org/gitlab-ce/issues/65504)). +1. If `needs:` refers to a job that is marked as `parallel:`. + the current job will depend on all parallel jobs created. +1. `needs:` is similar to `dependencies:` in that it needs to use jobs from + prior stages, meaning it is impossible to create circular + dependencies or depend on jobs in the current stage (see [gitlab-ce#65505](https://gitlab.com/gitlab-org/gitlab-ce/issues/65505)). +1. Related to the above, stages must be explicitly defined for all jobs + that have the keyword `needs:` or are referred to by one. +1. For self-managed users, the feature must be turned on using the `ci_dag_support` + feature flag. The `ci_dag_limit_needs` option, if set, will limit the number of + jobs that a single job can need to `50`. If unset, the limit is `5`. + ### `coverage` > [Introduced][ce-7447] in GitLab 8.17. @@ -1741,14 +1983,41 @@ sequentially from `job_name 1/N` to `job_name N/N`. For every job, `CI_NODE_INDEX` and `CI_NODE_TOTAL` [environment variables](../variables/README.md#predefined-environment-variables) are set. -A simple example: +Marking a job to be run in parallel requires only a simple addition to your configuration file: + +```diff + test: + script: rspec ++ parallel: 5 +``` +TIP: **Tip:** +Parallelize tests suites across parallel jobs. +Different languages have different tools to facilitate this. + +A simple example using [Sempahore Test Boosters](https://github.com/renderedtext/test-boosters) and RSpec to run some Ruby tests: + +```ruby +# Gemfile +source 'https://rubygems.org' + +gem 'rspec' +gem 'semaphore_test_boosters' +``` ```yaml test: - script: rspec - parallel: 5 + parallel: 3 + script: + - bundle + - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL ``` +CAUTION: **Caution:** +Please be aware that semaphore_test_boosters reports usages statistics to the author. + +You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec +job split into three separate jobs. + ### `trigger` **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. @@ -2221,10 +2490,10 @@ spinach: script: rake spinach ``` -It's also possible to use multiple parents for `extends`. -The algorithm used for merge is "closest scope wins", so keys -from the last member will always shadow anything defined on other levels. -For example: +In GitLab 12.0 and later, it's also possible to use multiple parents for +`extends`. The algorithm used for merge is "closest scope wins", so +keys from the last member will always shadow anything defined on other +levels. For example: ```yaml .only-important: @@ -2411,20 +2680,20 @@ There are three possible values: `none`, `normal`, and `recursive`: - `normal` means that only the top-level submodules will be included. It is equivalent to: - ``` - git submodule sync - git submodule update --init - ``` + ``` + git submodule sync + git submodule update --init + ``` - `recursive` means that all submodules (including submodules of submodules) will be included. This feature needs Git v1.8.1 and later. When using a GitLab Runner with an executor not based on Docker, make sure the Git version meets that requirement. It is equivalent to: - ``` - git submodule sync --recursive - git submodule update --init --recursive - ``` + ``` + git submodule sync --recursive + git submodule update --init --recursive + ``` Note that for this feature to work correctly, the submodules must be configured (in `.gitmodules`) with either: @@ -2550,18 +2819,39 @@ You can set it globally or per-job in the [`variables`](#variables) section. The following parameters are deprecated. -### `types` +### Globally-defined `types` CAUTION: **Deprecated:** `types` is deprecated, and could be removed in a future release. Use [`stages`](#stages) instead. -### `type` +### Job-defined `type` CAUTION: **Deprecated:** `type` is deprecated, and could be removed in one of the future releases. Use [`stage`](#stage) instead. +### Globally-defined `image`, `services`, `cache`, `before_script`, `after_script` + +Defining `image`, `services`, `cache`, `before_script`, and +`after_script` globally is deprecated. Support could be removed +from a future release. + +Use [`default:`](#setting-default-parameters) instead. For example: + +```yaml +default: + image: ruby:2.5 + services: + - docker:dind + cache: + paths: [vendor/] + before_script: + - bundle install --path vendor/ + after_script: + - rm -rf tmp/ +``` + ## Custom build directories > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1267) in Gitlab Runner 11.10 @@ -2644,7 +2934,7 @@ variables: The value of `GIT_CLONE_PATH` is expanded once into `$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure -because `$CI_BUILDS_DIR` is not expanded. +because `$CI_BUILDS_DIR` is not expanded. ## Special YAML features @@ -2829,7 +3119,8 @@ Alternatively, one can pass the `ci.skip` [Git push option][push-option] if using Git 2.10 or newer: ```sh -git push -o ci.skip +git push --push-option=ci.skip # using git 2.10+ +git push -o ci.skip # using git 2.18+ ``` <!-- ## Troubleshooting diff --git a/doc/customization/branded_login_page.md b/doc/customization/branded_login_page.md index b892f59d777..afcc2b71284 100644 --- a/doc/customization/branded_login_page.md +++ b/doc/customization/branded_login_page.md @@ -1,19 +1,38 @@ -# Changing the appearance of the login page +--- +type: howto +--- -GitLab offers a way to put your company's identity on the login page of your GitLab server and make it a branded login page. +# Changing the logo and description on the login page -By default, the page shows the GitLab logo and description. +You can customize the login page of your GitLab server to show the logo and +description of your organization. + +By default, the page shows the GitLab logo and description:  -## Changing the appearance of the login page +To customize the login page: -Navigate to the **Admin** area and go to the **Appearance** page. +1. Navigate to the **Admin** area and go to the **Appearance** page. +1. Fill in your desired Title and Description. You can also choose an image file + of the logo for your organization. -Fill in the required details like Title, Description and upload the company logo. +  - +1. Save your changes. -After saving the page, your GitLab login page will have the details you filled in: +Your GitLab login page will display the details you provided:  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/customization/branded_page_and_email_header.md b/doc/customization/branded_page_and_email_header.md index 9a0f0b382fa..370c1461d30 100644 --- a/doc/customization/branded_page_and_email_header.md +++ b/doc/customization/branded_page_and_email_header.md @@ -1,15 +1,37 @@ -# Changing the logo on the overall page and email header +--- +type: howto +--- -Navigate to the **Admin** area and go to the **Appearance** page. +# Changing the navigation bar and email header logo -Upload the custom logo (**Header logo**) in the section **Navigation bar**. +You can customize the logo that appears in email headers and in the navigation +bar on pages that are displayed by your GitLab server. - +1. Navigate to the **Admin** area and go to the **Appearance** page, then locate + the **Navigation bar** section. +1. For the **Header Logo**, choose an image file of the logo for your + organization. -After saving the page, your GitLab navigation bar will contain the custom logo: +  + +1. Save your changes. + +Your GitLab navigation bar will display the custom logo:  -The GitLab pipeline emails will also have the custom logo: +The GitLab pipeline emails will also display the custom logo:  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/customization/favicon.md b/doc/customization/favicon.md index 45a18159b5e..dbde6e8c03b 100644 --- a/doc/customization/favicon.md +++ b/doc/customization/favicon.md @@ -1,16 +1,37 @@ +--- +type: howto +--- + # Changing the favicon > [Introduced][ce-14497] in GitLab 11.0. [ce-14497]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14497 -Navigate to the **Admin** area and go to the **Appearance** page. +You can customize the favicon (the icon displayed in your web browser's +address bar and web page tabs) for your GitLab server. + +1. Navigate to the **Admin** area and go to the **Appearance** page, then + locate the **Favicon** section. +1. Upload an image file of your favicon. -Upload the custom favicon (**Favicon**) in the section **Favicon**. +  - +1. Save your changes. -After saving the page, the new favicon will be shown in the browser. The main -favicon as well as the CI status icons will show the custom icon: +Your new favicon will display in the browser. The main favicon and the CI +status icons will show the custom icon:  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/customization/help_message.md b/doc/customization/help_message.md index c2e592d03bf..a4d8f295750 100644 --- a/doc/customization/help_message.md +++ b/doc/customization/help_message.md @@ -1,13 +1,36 @@ -# GitLab Help custom text +--- +type: howto +--- -In larger organizations it is useful to have information about who has the responsibility of maintaining the company GitLab server. +# Customizing the 'Help' and login page messages -1. Navigate to the admin area, click on **Preferences** and expand **Help page**. +In large organizations, it is useful to have information about who maintains +the company GitLab server. You can customize and display this information on +the GitLab login page and on the GitLab server's `/help` page. -1. Under **Help text** fill in the required information about the person(s) administering GitLab or any other information relevant to your needs. +1. Navigate to the **Admin** area, then click on **Preferences** and expand + **Help page**. +1. Under **Help page text**, fill in the required information about the + person(s) administering GitLab. This text can also contain any other + information that you wish to display to users. -  +  -1. After saving the page this information will be shown on the GitLab login page and on the GitLab `/help` page (e.g., <https://gitlab.com/help>). +1. Save your changes. -  +The information you entered will be shown on the GitLab login page and on the +GitLab `/help` page (e.g., <https://gitlab.com/help>). + + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/customization/help_message/help_text.png b/doc/customization/help_message/help_text.png Binary files differindex 99697a106bf..5fcabcdb757 100644 --- a/doc/customization/help_message/help_text.png +++ b/doc/customization/help_message/help_text.png diff --git a/doc/customization/index.md b/doc/customization/index.md index 0198059297f..f17a2d80e2c 100644 --- a/doc/customization/index.md +++ b/doc/customization/index.md @@ -1,18 +1,18 @@ --- +type: index description: Learn how to customize GitLab's appearance for self-managed installations. --- # Customizing GitLab's appearance **(CORE ONLY)** -For GitLab self-managed instances, it's possible to customize -a few pages. +For GitLab self-managed instances, you can customize the page logo, +email headers, favicon, and several other aspects of GitLab's appearance. -Read through the following documents to adjust GitLab's -look and feel to meet your needs: +The following pages explain how to customize the appearance of your instance: -- [Custom login page](branded_login_page.md) -- [Custom header and email logo](branded_page_and_email_header.md) -- [Custom favicon](favicon.md) -- [Libravatar](libravatar.md) -- [New project page](new_project_page.md) -- [Custom `/help` message](help_message.md)
\ No newline at end of file +- [Changing the logo and description on the login page](branded_login_page.md) +- [Changing the navigation bar and email header logo](branded_page_and_email_header.md) +- [Changing the favicon](favicon.md) +- [Customizing the new project page](new_project_page.md) +- [Customizing the `/help` and login page messages](help_message.md) +- [Using the Libravatar service with GitLab](libravatar.md) diff --git a/doc/customization/issue_and_merge_request_template.md b/doc/customization/issue_and_merge_request_template.md index adaa120a37e..ebf711e105b 100644 --- a/doc/customization/issue_and_merge_request_template.md +++ b/doc/customization/issue_and_merge_request_template.md @@ -1,5 +1,5 @@ --- -redirect_to: '../user/project/description_templates.md#setting-a-default-template-for-issues-and-merge-requests--starter' +redirect_to: '../user/project/description_templates.md#setting-a-default-template-for-merge-requests-and-issues--starter' --- -This document was moved to [description_templates](../user/project/description_templates.md#setting-a-default-template-for-issues-and-merge-requests--starter). +This document was moved to [description_templates](../user/project/description_templates.md#setting-a-default-template-for-merge-requests-and-issues--starter). diff --git a/doc/customization/libravatar.md b/doc/customization/libravatar.md index e618f3be2fe..6d96528f760 100644 --- a/doc/customization/libravatar.md +++ b/doc/customization/libravatar.md @@ -1,14 +1,20 @@ -# Use Libravatar service with GitLab +--- +type: howto +--- -GitLab by default supports [Gravatar](https://gravatar.com) avatar service. -Libravatar is a service which delivers your avatar (profile picture) to other websites and their API is -[heavily based on gravatar](https://wiki.libravatar.org/api/). +# Using the Libravatar service with GitLab -This means that it is not complicated to switch to Libravatar avatar service or even self hosted Libravatar server. +GitLab by default supports the [Gravatar](https://gravatar.com) avatar service. + +Libravatar is another service that delivers your avatar (profile picture) to +other websites. The Libravatar API is +[heavily based on gravatar](https://wiki.libravatar.org/api/), so you can +easily switch to the Libravatar avatar service or even a self-hosted Libravatar +server. ## Configuration -In [gitlab.yml gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122) set +In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab-ce/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set the configuration options as follows: ### For HTTP @@ -29,50 +35,67 @@ the configuration options as follows: ssl_url: "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" ``` -### Self-hosted +### Self-hosted Libravatar server -If you are [running your own libravatar service](https://wiki.libravatar.org/running_your_own/) the URL will be different in the configuration -but the important part is to provide the same placeholders so GitLab can parse the URL correctly. +If you are [running your own libravatar service](https://wiki.libravatar.org/running_your_own/), +the URL will be different in the configuration, but you must provide the same +placeholders so GitLab can parse the URL correctly. -For example, you host a service on `http://libravatar.example.com` the `plain_url` you need to supply in `gitlab.yml` is +For example, you host a service on `http://libravatar.example.com` and the +`plain_url` you need to supply in `gitlab.yml` is `http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` -### Omnibus-gitlab example +### Omnibus GitLab example In `/etc/gitlab/gitlab.rb`: -#### For http +#### For HTTP ```ruby gitlab_rails['gravatar_enabled'] = true gitlab_rails['gravatar_plain_url'] = "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" ``` -#### For https +#### For HTTPS ```ruby gitlab_rails['gravatar_enabled'] = true gitlab_rails['gravatar_ssl_url'] = "https://seccdn.libravatar.org/avatar/%{hash}?s=%{size}&d=identicon" ``` -Run `sudo gitlab-ctl reconfigure` for changes to take effect. +Then run `sudo gitlab-ctl reconfigure` for the changes to take effect. ## Default URL for missing images -[Libravatar supports different sets](https://wiki.libravatar.org/api/) of `missing images` for emails not found on the Libravatar service. - -In order to use a different set other than `identicon`, replace `&d=identicon` portion of the URL with another supported set. -For example, you can use `retro` set in which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` +[Libravatar supports different sets](https://wiki.libravatar.org/api/) of +missing images for user email addresses that are not found on the Libravatar +service. -## Usage examples +In order to use a set other than `identicon`, replace the `&d=identicon` +portion of the URL with another supported set. +For example, you can use the `retro` set, in which case the URL would look like: +`plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` -### For Microsoft Office 365 +## Usage examples for Microsoft Office 365 -If your users are Office 365-users, the "GetPersonaPhoto" service can be used. Note that this service requires login, so this use case is -most useful in a corporate installation, where all users have access to Office 365. +If your users are Office 365 users, the `GetPersonaPhoto` service can be used. +Note that this service requires a login, so this use case is most useful in a +corporate installation where all users have access to Office 365. ```ruby gitlab_rails['gravatar_plain_url'] = 'http://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' ``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/customization/new_project_page.md b/doc/customization/new_project_page.md index 148bf9512c6..43b95a76d08 100644 --- a/doc/customization/new_project_page.md +++ b/doc/customization/new_project_page.md @@ -1,20 +1,38 @@ +--- +type: howto +--- + # Customizing the new project page -It is possible to add a markdown-formatted message to your GitLab -new project page. +You can add a markdown-formatted message to your GitLab new project page. By default, the new project page shows a sidebar with general information: - + + +To customize the information in the sidebar: + +1. Navigate to the **Admin** area and go to the **Appearance** page, then + locate the **New project pages** section. +1. Fill in your new project project guidelines: + +  -## Changing the appearance of the new project page +1. Save the page. -Navigate to the **Admin** area and go to the **Appearance** page. +Your new project page will show the customized guidelines in the sidebar, below +the general information: -Fill in your project guidelines: + - +<!-- ## Troubleshooting -After saving the page, your new project page will show the guidelines in the sidebar, below the general information: +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. - +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/customization/system_header_and_footer_messages.md b/doc/customization/system_header_and_footer_messages.md index 15830be4e8a..bd2de3e201c 100644 --- a/doc/customization/system_header_and_footer_messages.md +++ b/doc/customization/system_header_and_footer_messages.md @@ -8,7 +8,7 @@ Navigate to the **Admin** area and go to the **Appearance** page. Under **System header and footer** insert your header message and/or footer message. Both background and font color of the header and footer are customizable. -You can also apply the header and footer messages to gitlab emails, +You can also apply the header and footer messages to GitLab emails, by checking the **Enable header and footer in emails** checkbox. Note that color settings will only be applied within the app interface and not to emails diff --git a/doc/customization/system_header_and_footer_messages/appearance.png b/doc/customization/system_header_and_footer_messages/appearance.png Binary files differindex fd315bb6c07..d5a66bcb9f1 100644 --- a/doc/customization/system_header_and_footer_messages/appearance.png +++ b/doc/customization/system_header_and_footer_messages/appearance.png diff --git a/doc/development/README.md b/doc/development/README.md index 1566173992a..3912a828dec 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -3,9 +3,9 @@ comments: false description: 'Learn how to contribute to GitLab.' --- -# GitLab development guides +# Contributor and Development Docs -## Get started! +## Get started - Set up GitLab's development environment with [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/README.md) - [GitLab contributing guide](contributing/index.md) @@ -17,12 +17,13 @@ description: 'Learn how to contribute to GitLab.' - [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/PROCESS.md) - [Generate a changelog entry with `bin/changelog`](changelog.md) - [Code review guidelines](code_review.md) for reviewing code and having code reviewed +- [Database review guidelines](database_review.md) for reviewing database-related changes and complex SQL queries - [Automatic CE->EE merge](automatic_ce_ee_merge.md) - [Guidelines for implementing Enterprise Edition features](ee_features.md) - [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) - [Requesting access to Chatops on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLabbers) -## UX and frontend guides +## UX and Frontend guides - [GitLab Design System](https://design.gitlab.com/) for building GitLab with existing CSS styles and elements - [Frontend guidelines](fe_guide/index.md) @@ -63,6 +64,8 @@ description: 'Learn how to contribute to GitLab.' - [Routing](routing.md) - [Repository mirroring](repository_mirroring.md) - [Git LFS](lfs.md) +- [Developing against interacting components or features](interacting_components.md) +- [File uploads](uploads.md) ## Performance guides @@ -111,6 +114,11 @@ description: 'Learn how to contribute to GitLab.' - [Database helper modules](database_helpers.md) - [Code comments](code_comments.md) +## Case studies + +- [Database case study: Filtering by label](filtering_by_label.md) +- [Database case study: Namespaces storage statistics](namespaces_storage_statistics.md) + ## Integration guides - [Jira Connect app](integrations/jira_connect.md) @@ -144,6 +152,10 @@ description: 'Learn how to contribute to GitLab.' - [Go Guidelines](go_guide/index.md) +## Shell Scripting guides + +- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md) + ## Other GitLab Development Kit (GDK) guides - [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/auto_devops.md) diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index c83a0427c98..7569ccc04c1 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -424,12 +424,8 @@ Will generate a field called `mergeRequestSetWip` that ### Authorizing resources -To authorize resources inside a mutation, we can include the -`Gitlab::Graphql::Authorize::AuthorizeResource` concern in the -mutation. - -This allows us to provide the required abilities on the mutation like -this: +To authorize resources inside a mutation, we first provide the required + abilities on the mutation like this: ```ruby module Mutations diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 0866d3baeeb..61576236c96 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -51,7 +51,7 @@ allowed. – <https://github.com/ruby-grape/grape#declared> -### Exclude params from parent namespaces! +### Exclude params from parent namespaces > By default `declared(params)`includes parameters that were defined in all parent namespaces. @@ -64,7 +64,7 @@ In most cases you will want to exclude params from the parent namespaces: declared(params, include_parent_namespaces: false) ``` -### When to use `declared(params)`? +### When to use `declared(params)` You should always use `declared(params)` when you pass the params hash as arguments to a method call. diff --git a/doc/development/architecture.md b/doc/development/architecture.md index b645a72567c..2adca2dae28 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -12,21 +12,21 @@ New versions of GitLab are released in stable branches and the master branch is For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/tree/master#gitlab-release-process). -Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. +Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. ## Components A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs. -We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/). +We also support deploying GitLab on Kubernetes using our [GitLab Helm chart](https://docs.gitlab.com/charts/). -The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. +The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare Git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. +When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving Git objects. -The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories through Gitaly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access. +The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. -Gitaly executes git operations from gitlab-shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). +Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/). @@ -130,7 +130,7 @@ Component statuses are linked to configuration documentation for each component. | [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [✅][nginx-omnibus] | [✅][nginx-charts] | [⚙][nginx-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][nginx-source] | ❌ | CE & EE | | [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [✅][unicorn-omnibus] | [✅][unicorn-charts] | [✅][unicorn-charts] | [✅](../user/gitlab_com/index.md#unicorn) | [⚙][unicorn-source] | [✅][gitlab-yml] | CE & EE | | [Sidekiq](#sidekiq) | Background jobs processor | [✅][sidekiq-omnibus] | [✅][sidekiq-charts] | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](../user/gitlab_com/index.md#sidekiq) | [✅][gitlab-yml] | [✅][gitlab-yml] | CE & EE | -| [Gitaly](#gitaly) | Git RPC service for handling all git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE | +| [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE | | [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][workhorse-source] | ✅ | CE & EE | | [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][shell-source] | [✅][gitlab-yml] | CE & EE | | [GitLab Pages](#gitlab-pages) | Hosts static websites | [⚙][pages-omnibus] | [❌][pages-charts] | [❌][pages-charts] | [✅](../user/gitlab_com/index.md#gitlab-pages) | [⚙][pages-source] | [⚙][pages-gdk] | CE & EE | @@ -147,7 +147,7 @@ Component statuses are linked to configuration documentation for each component. | [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [✅][redis-exporter-omnibus] | [✅][redis-exporter-charts] | [✅][redis-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [✅][postgres-exporter-omnibus] | [✅][postgres-exporter-charts] | [✅][postgres-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | [⚙][pgbouncer-exporter-omnibus] | [❌][pgbouncer-exporter-charts] | [❌][pgbouncer-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | -| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [✅][gitab-monitor-charts] | [✅][gitab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [✅][gitlab-monitor-charts] | [✅][gitlab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | [✅][node-exporter-omnibus] | [❌][node-exporter-charts] | [❌][node-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | | [Mattermost](#mattermost) | Open-source Slack alternative | [⚙][mattermost-omnibus] | [⤓][mattermost-charts] | [⤓][mattermost-charts] | [⤓](../user/project/integrations/mattermost.md) | ❌ | ❌ | CE & EE | | [MinIO](#minio) | Object storage service | [⤓][minio-omnibus] | [✅][minio-charts] | [✅][minio-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | ❌ | [⚙][minio-gdk] | CE & EE | @@ -185,7 +185,7 @@ GitLab can be considered to have two layers from a process perspective: - Layer: Monitoring - Process: `alertmanager` -[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. +[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue #45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. #### Certificate management @@ -223,12 +223,12 @@ Elasticsearch is a distributed RESTful search engine built for the cloud. Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). -#### Gitlab Geo +#### GitLab Geo - Configuration: [Omnibus][geo-omnibus], [Charts][geo-charts], [GDK][geo-gdk] - Layer: Core Service (Processor) -#### Gitlab Monitor +#### GitLab Monitor - [Project page](https://gitlab.com/gitlab-org/gitlab-monitor) - Configuration: [Omnibus][gitlab-monitor-omnibus], [Charts][gitlab-monitor-charts] @@ -237,7 +237,7 @@ Gitaly is a service designed by GitLab to remove our need for NFS for Git storag GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor). -#### Gitlab Pages +#### GitLab Pages - Configuration: [Omnibus][pages-omnibus], [Charts][pages-charts], [Source][pages-source], [GDK][pages-gdk] - Layer: Core Service (Processor) @@ -246,7 +246,7 @@ GitLab Pages is a feature that allows you to publish static websites directly fr You can use it either for personal or business websites, such as portfolios, documentation, manifestos, and business presentations. You can also attribute any license to your content. -#### Gitlab Runner +#### GitLab Runner - [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md) - Configuration: [Omnibus][runner-omnibus], [Charts][runner-charts], [Source][runner-source], [GDK][runner-gdk] @@ -256,7 +256,7 @@ GitLab Runner runs tests and sends the results to GitLab. GitLab CI is the open-source continuous integration service included with GitLab that coordinates the testing. The old name of this project was GitLab CI Multi Runner but please use "GitLab Runner" (without CI) from now on. -#### Gitlab Shell +#### GitLab Shell - [Project page](https://gitlab.com/gitlab-org/gitlab-shell/blob/master/README.md) - Configuration: [Omnibus][shell-omnibus], [Charts][shell-charts], [Source][shell-source], [GDK][gitlab-yml] @@ -264,7 +264,7 @@ GitLab CI is the open-source continuous integration service included with GitLab [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) is a program designed at GitLab to handle ssh-based `git` sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. -#### Gitlab Workhorse +#### GitLab Workhorse - [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) - Configuration: [Omnibus][workhorse-omnibus], [Charts][workhorse-charts], [Source][workhorse-source] @@ -475,7 +475,7 @@ It's important to understand the distinction as some processes are used in both When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: - nginx - Acts as our first line reverse proxy. -- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. +- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. - unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. - Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. @@ -493,13 +493,13 @@ TODO ## System Layout -When referring to `~git` in the pictures it means the home directory of the git user which is typically `/home/git`. +When referring to `~git` in the pictures it means the home directory of the Git user which is typically `/home/git`. GitLab is primarily installed within the `/home/git` user home directory as `git` user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable). The bare repositories are located in `/home/git/repositories`. GitLab is a ruby on rails application so the particulars of the inner workings can be learned by studying how a ruby on rails application works. -To serve repositories over SSH there's an add-on application called gitlab-shell which is installed in `/home/git/gitlab-shell`. +To serve repositories over SSH there's an add-on application called GitLab Shell which is installed in `/home/git/gitlab-shell`. ### Installation Folder Summary @@ -511,11 +511,19 @@ To summarize here's the [directory structure of the `git` user home directory](. ps aux | grep '^git' ``` -GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process). +GitLab has several components to operate. It requires a persistent database +(PostgreSQL) and redis database, and uses Apache httpd or Nginx to proxypass +Unicorn. All these components should run as different system users to GitLab +(e.g., `postgres`, `redis` and `www-data`, instead of `git`). + +As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server +running on port `8080` by default). Under the GitLab user there are normally 4 +processes: `unicorn_rails master` (1 process), `unicorn_rails worker` +(2 processes), `sidekiq` (1 process). ### Repository access -Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by gitlab-shell (previously explained). +Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by GitLab Shell (previously explained). ## Troubleshooting @@ -523,28 +531,28 @@ See the README for more information. ### Init scripts of the services -The GitLab init script starts and stops Unicorn and Sidekiq. +The GitLab init script starts and stops Unicorn and Sidekiq: ``` /etc/init.d/gitlab Usage: service gitlab {start|stop|restart|reload|status} ``` -Redis (key-value store/non-persistent database) +Redis (key-value store/non-persistent database): ``` /etc/init.d/redis Usage: /etc/init.d/redis {start|stop|status|restart|condrestart|try-restart} ``` -SSH daemon +SSH daemon: ``` /etc/init.d/sshd Usage: /etc/init.d/sshd {start|stop|restart|reload|force-reload|condrestart|try-restart|status} ``` -Web server (one of the following) +Web server (one of the following): ``` /etc/init.d/httpd @@ -554,54 +562,46 @@ $ /etc/init.d/nginx Usage: nginx {start|stop|restart|reload|force-reload|status|configtest} ``` -Persistent database (one of the following) +Persistent database: ``` -/etc/init.d/mysqld -Usage: /etc/init.d/mysqld {start|stop|status|restart|condrestart|try-restart|reload|force-reload} - $ /etc/init.d/postgresql Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..] ``` ### Log locations of the services -gitlabhq (includes Unicorn and Sidekiq logs) +gitlabhq (includes Unicorn and Sidekiq logs): -- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `githost.log` and `unicorn.stderr.log` normally. +- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `git_json.log` and `unicorn.stderr.log` normally. -gitlab-shell +GitLab Shell: - `/home/git/gitlab-shell/gitlab-shell.log` -ssh +SSH: - `/var/log/auth.log` auth log (on Ubuntu). - `/var/log/secure` auth log (on RHEL). -nginx +nginx: - `/var/log/nginx/` contains error and access logs. -Apache httpd +Apache httpd: - [Explanation of Apache logs](https://httpd.apache.org/docs/2.2/logs.html). - `/var/log/apache2/` contains error and output logs (on Ubuntu). - `/var/log/httpd/` contains error and output logs (on RHEL). -redis +Redis: - `/var/log/redis/redis.log` there are also log-rotated logs there. -PostgreSQL +PostgreSQL: - `/var/log/postgresql/*` -MySQL - -- `/var/log/mysql/*` -- `/var/log/mysql.*` - ### GitLab specific config files GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include: @@ -610,7 +610,7 @@ GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly - `unicorn.rb` - Unicorn web server settings. - `database.yml` - Database connection settings. -gitlab-shell has a configuration file at `/home/git/gitlab-shell/config.yml`. +GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. ### Maintenance Tasks diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index 423b35a9e3a..c2700461467 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -171,6 +171,19 @@ Now, every time you create an MR for CE and EE: job failed, you are required to submit the EE MR so that you can fix the conflicts in EE before merging your changes into CE. +## How we run the Automatic CE->EE merge at GitLab + +At GitLab, we use the [Merge Train](https://gitlab.com/gitlab-org/merge-train) +project to keep our [GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee) +repository updated with commits from +[GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce). + +We have a mirror of the [Merge Train](https://gitlab.com/gitlab-org/merge-train) +project [configured](https://ops.gitlab.net/gitlab-org/merge-train) to run an +automatic CE->EE merge job every twenty minutes as a scheduled CI job. The +[configured](https://ops.gitlab.net/gitlab-org/merge-train) Merge Train project +is only accessible to authorized GitLab staff. + ## FAQ ### How does automatic merging work? @@ -187,7 +200,7 @@ code. ### Why merge automatically? As we work towards continuous deployments and a single repository for both CE -and EE, we need to first make sure that all CE changes make their way into CE as +and EE, we need to first make sure that all CE changes make their way into EE as fast as possible. Past experiences and data have shown that periodic CE to EE merge requests do not scale, and often take a very long time to complete. For example, [in this diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 642dac614c7..3fd95537eaa 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -294,7 +294,7 @@ to migrate you database down and up, which can result in other background migrations being called. That means that using `spy` test doubles with `have_received` is encouraged, instead of using regular test doubles, because your expectations defined in a `it` block can conflict with what is being -called in RSpec hooks. See [gitlab-org/gitlab-ce#35351][issue-rspec-hooks] +called in RSpec hooks. See [issue #35351][issue-rspec-hooks] for more details. ## Best practices diff --git a/doc/development/build_test_package.md b/doc/development/build_test_package.md index c5f6adfeaeb..21891f70d73 100644 --- a/doc/development/build_test_package.md +++ b/doc/development/build_test_package.md @@ -3,7 +3,7 @@ While developing a new feature or modifying an existing one, it is helpful if an installable package (or a docker image) containing those changes is available for testing. For this very purpose, a manual job is provided in the GitLab CI/CD -pipeline that can be used to trigger a pipeline in the omnibus-gitlab repository +pipeline that can be used to trigger a pipeline in the Omnibus GitLab repository that will create: - A deb package for Ubuntu 16.04, available as a build artifact, and @@ -12,7 +12,7 @@ that will create: (images titled `gitlab-ce` and `gitlab-ee` respectively and image tag is the commit which triggered the pipeline). -When you push a commit to either the gitlab-ce or gitlab-ee project, the +When you push a commit to either the GitLab CE or GitLab EE project, the pipeline for that commit will have a `build-package` manual action you can trigger. @@ -30,9 +30,9 @@ branch `0-1-stable`, modify the content of `GITALY_SERVER_VERSION` to `0-1-stable` and push the commit. This will create a manual job that can be used to trigger the build. -## Specifying the branch in omnibus-gitlab repository +## Specifying the branch in Omnibus GitLab repository -In scenarios where a configuration change is to be introduced and omnibus-gitlab +In scenarios where a configuration change is to be introduced and Omnibus GitLab repository already has the necessary changes in a specific branch, you can build a package against that branch through an environment variable named `OMNIBUS_BRANCH`. To do this, specify that environment variable with the name of diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 3ed586f07e9..814624c7586 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -35,6 +35,7 @@ the `author` field. GitLab team members **should not**. - Any user-facing change **should** have a changelog entry. Example: "GitLab now uses system fonts for all text." +- Any docs-only changes **should not** have a changelog entry. - Any change behind a feature flag **should not** have a changelog entry. The entry should be added [in the merge request removing the feature flags](feature_flags/development.md). - A fix for a regression introduced and then fixed in the same release (i.e., fixing a bug introduced during a monthly release candidate) **should not** @@ -129,6 +130,7 @@ merge_request: author: type: ``` + If you're working on the GitLab EE repository, the entry will be added to `ee/changelogs/unreleased/` instead. diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 403a5b21827..961520db7d8 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -15,23 +15,19 @@ Currently, there are four endpoints for simulating the following conditions: ## Enabling chaos endpoints -For obvious reasons, these endpoints are not enabled by default. They can be enabled by setting the `GITLAB_ENABLE_CHAOS_ENDPOINTS` environment variable to `1`. - -For example, if you're using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) this can be done with the following command: - -```bash -GITLAB_ENABLE_CHAOS_ENDPOINTS=1 gdk run -``` - -## Securing the chaos endpoints +For obvious reasons, these endpoints are not enabled by default on `production`. +They are enabled by default on **development** environments. DANGER: **Danger:** -It is highly recommended that you secure access to the chaos endpoints using a secret token. This is recommended when enabling these endpoints locally and essential when running in a staging or other shared environment. You should not enable them in production unless you absolutely know what you're doing. +It is required that you secure access to the chaos endpoints using a secret token. +You should not enable them in production unless you absolutely know what you're doing. -A secret token can be set through the `GITLAB_CHAOS_SECRET` environment variable. For example, when using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) this can be done with the following command: +A secret token can be set through the `GITLAB_CHAOS_SECRET` environment variable. +For example, when using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) +this can be done with the following command: ```bash -GITLAB_ENABLE_CHAOS_ENDPOINTS=1 GITLAB_CHAOS_SECRET=secret gdk run +GITLAB_CHAOS_SECRET=secret gdk run ``` Replace `secret` with your own secret token. @@ -40,6 +36,10 @@ Replace `secret` with your own secret token. Once you have enabled the chaos endpoints and restarted the application, you can start testing using the endpoints. +By default, when invoking a chaos endpoint, the web worker process which receives the request will handle it. This means, for example, that if the Kill +operation is invoked, the Puma or Unicorn worker process handling the request will be killed. To test these operations in Sidekiq, the `async` parameter on +each endpoint can be set to `true`. This will run the chaos process in a Sidekiq worker. + ## Memory leaks To simulate a memory leak in your application, use the `/-/chaos/leakmem` endpoint. @@ -51,54 +51,88 @@ The memory is not retained after the request finishes. Once the request has comp GET /-/chaos/leakmem GET /-/chaos/leakmem?memory_mb=1024 GET /-/chaos/leakmem?memory_mb=1024&duration_s=50 +GET /-/chaos/leakmem?memory_mb=1024&duration_s=50&async=true ``` -| Attribute | Type | Required | Description | -| ------------ | ------- | -------- | ---------------------------------------------------------------------------------- | -| `memory_mb` | integer | no | How much memory, in MB, should be leaked. Defaults to 100MB. | -| `duration_s` | integer | no | Minimum duration, in seconds, that the memory should be retained. Defaults to 30s. | +| Attribute | Type | Required | Description | +| ------------ | ------- | -------- | ------------------------------------------------------------------------------------ | +| `memory_mb` | integer | no | How much memory, in MB, should be leaked. Defaults to 100MB. | +| `duration_s` | integer | no | Minimum duration_s, in seconds, that the memory should be retained. Defaults to 30s. | +| `async` | boolean | no | Set to true to leak memory in a Sidekiq background worker process | ```bash curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10 --header 'X-Chaos-Secret: secret' +curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=secret ``` ## CPU spin This endpoint attempts to fully utilise a single core, at 100%, for the given period. -Depending on your rack server setup, your request may timeout after a predermined period (normally 60 seconds). +Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds). If you're using Unicorn, this is done by killing the worker process. ``` -GET /-/chaos/cpuspin -GET /-/chaos/cpuspin?duration_s=50 +GET /-/chaos/cpu_spin +GET /-/chaos/cpu_spin?duration_s=50 +GET /-/chaos/cpu_spin?duration_s=50&async=true ``` | Attribute | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------- | -| `duration_s` | integer | no | Duration, in seconds, that the core will be utilised. Defaults to 30s | +| `duration_s` | integer | no | Duration, in seconds, that the core will be utilized. Defaults to 30s | +| `async` | boolean | no | Set to true to consume CPU in a Sidekiq background worker process | + +```bash +curl http://localhost:3000/-/chaos/cpu_spin?duration_s=60 --header 'X-Chaos-Secret: secret' +curl http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret +``` + +## DB spin + +This endpoint attempts to fully utilise a single core, and interleave it with DB request, for the given period. +This endpoint can be used to model yielding execution to another threads when running concurrently. + +Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds). +If you're using Unicorn, this is done by killing the worker process. + +``` +GET /-/chaos/db_spin +GET /-/chaos/db_spin?duration_s=50 +GET /-/chaos/db_spin?duration_s=50&async=true +``` + +| Attribute | Type | Required | Description | +| ------------ | ------- | -------- | --------------------------------------------------------------------------- | +| `interval_s` | float | no | Interval, in seconds, for every DB request. Defaults to 1s | +| `duration_s` | integer | no | Duration, in seconds, that the core will be utilized. Defaults to 30s | +| `async` | boolean | no | Set to true to perform the operation in a Sidekiq background worker process | ```bash -curl http://localhost:3000/-/chaos/cpuspin?duration_s=60 --header 'X-Chaos-Secret: secret' +curl http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60 --header 'X-Chaos-Secret: secret' +curl http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60&token=secret ``` ## Sleep -This endpoint is similar to the CPU Spin endpoint but simulates off-processor activity, such as network calls to backend services. It will sleep for a given duration. +This endpoint is similar to the CPU Spin endpoint but simulates off-processor activity, such as network calls to backend services. It will sleep for a given duration_s. -As with the CPU Spin endpoint, this may lead to your request timing out if duration exceeds the configured limit. +As with the CPU Spin endpoint, this may lead to your request timing out if duration_s exceeds the configured limit. ``` GET /-/chaos/sleep GET /-/chaos/sleep?duration_s=50 +GET /-/chaos/sleep?duration_s=50&async=true ``` | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ---------------------------------------------------------------------- | | `duration_s` | integer | no | Duration, in seconds, that the request will sleep for. Defaults to 30s | +| `async` | boolean | no | Set to true to sleep in a Sidekiq background worker process | ```bash curl http://localhost:3000/-/chaos/sleep?duration_s=60 --header 'X-Chaos-Secret: secret' +curl http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret ``` ## Kill @@ -110,8 +144,14 @@ Since this endpoint uses the `KILL` signal, the worker is not given a chance to ``` GET /-/chaos/kill +GET /-/chaos/kill?async=true ``` +| Attribute | Type | Required | Description | +| ------------ | ------- | -------- | ---------------------------------------------------------------------- | +| `async` | boolean | no | Set to true to kill a Sidekiq background worker process | + ```bash curl http://localhost:3000/-/chaos/kill --header 'X-Chaos-Secret: secret' +curl http://localhost:3000/-/chaos/kill?token=secret ``` diff --git a/doc/development/code_review.md b/doc/development/code_review.md index e60800f1ab7..b7d74b17eb3 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -45,9 +45,9 @@ page, with these behaviours: 1. It will not pick people whose [GitLab status](../user/profile/index.md#current-status) contains the string 'OOO'. -2. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) +1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. -3. It always picks the same reviewers and maintainers for the same +1. It always picks the same reviewers and maintainers for the same branch name (unless their OOO status changes, as in point 1). It removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so that it can be stable for backport branches. @@ -58,20 +58,21 @@ As described in the section on the responsibility of the maintainer below, you are recommended to get your merge request approved and merged by maintainer(s) from teams other than your own. - 1. If your merge request includes backend changes [^1], it must be - **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_backend)**. - 1. If your merge request includes database migrations or changes to expensive queries [^2], it must be - **approved by a [database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_database)**. - 1. If your merge request includes frontend changes [^1], it must be - **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_frontend)**. - 1. If your merge request includes UX changes [^1], it must be - **approved by a [UX team member][team]**. - 1. If your merge request includes adding a new JavaScript library [^1], it must be - **approved by a [frontend lead][team]**. - 1. If your merge request includes adding a new UI/UX paradigm [^1], it must be - **approved by a [UX lead][team]**. - 1. If your merge request includes a new dependency or a filesystem change, it must be - **approved by a [Distribution team member][team]**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/dev-backend/distribution/) for more details. +1. If your merge request includes backend changes [^1], it must be + **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_backend)**. +1. If your merge request includes database migrations or changes to expensive queries [^2], it must be + **approved by a [database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_database)**. + Read the [database review guidelines](database_review.md) for more details. +1. If your merge request includes frontend changes [^1], it must be + **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_frontend)**. +1. If your merge request includes UX changes [^1], it must be + **approved by a [UX team member][team]**. +1. If your merge request includes adding a new JavaScript library [^1], it must be + **approved by a [frontend lead][team]**. +1. If your merge request includes adding a new UI/UX paradigm [^1], it must be + **approved by a [UX lead][team]**. +1. If your merge request includes a new dependency or a filesystem change, it must be + **approved by a [Distribution team member][team]**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/dev-backend/distribution/) for more details. #### Security requirements @@ -207,9 +208,9 @@ first time. - Extract unrelated changes and refactorings into future merge requests/issues. - Seek to understand the reviewer's perspective. - Try to respond to every comment. -- The merge request author resolves only the discussions they have fully - addressed. If there's an open reply, an open discussion, a suggestion, - a question, or anything else, the discussion should be left to be resolved +- The merge request author resolves only the threads they have fully + addressed. If there's an open reply, an open thread, a suggestion, + a question, or anything else, the thread should be left to be resolved by the reviewer. - Push commits based on earlier rounds of feedback as isolated commits to the branch. Do not squash until the branch is ready to merge. Reviewers should be @@ -341,8 +342,7 @@ Enterprise Edition instance. This has some implications: - [Background migrations](background_migrations.md) run in Sidekiq, and should only be done for migrations that would take an extreme amount of time at GitLab.com scale. -1. **Sidekiq workers** - [cannot change in a backwards-incompatible way](sidekiq_style_guide.md#removing-or-renaming-queues): +1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq_style_guide.md#removing-or-renaming-queues): 1. Sidekiq queues are not drained before a deploy happens, so there will be workers in the queue from the previous version of GitLab. 1. If you need to change a method signature, try to do so across two releases, @@ -365,6 +365,31 @@ Enterprise Edition instance. This has some implications: 1. **Filesystem access** can be slow, so try to avoid [shared files](shared_files.md) when an alternative solution is available. +## Examples + +How code reviews are conducted can surprise new contributors. Here are some examples of code reviews that should help to orient you as to what to expect. + +**["Modify `DiffNote` to reuse it for Designs"](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13703):** +It contained everything from nitpicks around newlines to reasoning +about what versions for designs are, how we should compare them +if there was no previous version of a certain file (parent vs. +blank `sha` vs empty tree). + +**["Support multi-line suggestions"](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25211)**: +The MR itself consists of a collaboration between FE and BE, +and documenting comments from the author for the reviewer. +There's some nitpicks, some questions for information, and +towards the end, a security vulnerability. + +**["Allow multiple repositories per project"](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10251)**: +ZJ referred to the other projects (workhorse) this might impact, +suggested some improvements for consistency. And James' comments +helped us with overall code quality (using delegation, `&.` those +types of things), and making the code more robust. + +**["Support multiple assignees for merge requests"](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10161)**: +A good example of collaboration on an MR touching multiple parts of the codebase. Nick pointed out interesting edge cases, James Lopes also joined in raising concerns on import/export feature. + ### Credits Largely based on the [thoughtbot code review guide]. diff --git a/doc/development/contributing/community_roles.md b/doc/development/contributing/community_roles.md index 3296cb173d7..7d2d1b77a0e 100644 --- a/doc/development/contributing/community_roles.md +++ b/doc/development/contributing/community_roles.md @@ -1,4 +1,4 @@ -### Community members & roles +# Community members & roles GitLab community members and their privileges/responsibilities. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 910f9f4bf7a..b2e3ef7bf63 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -58,78 +58,33 @@ issue is labeled with a subject label corresponding to your expertise. Subject labels are always all-lowercase. -## Team labels +## Team labels -**Important**: Most of the team labels will be soon deprecated in favor of [Group labels](#group-labels). +**Important**: Most of the historical team labels (e.g. Manage, Plan etc.) are +now deprecated in favor of [Group labels](#group-labels) and [Stage labels](#stage-labels). Team labels specify what team is responsible for this issue. Assigning a team label makes sure issues get the attention of the appropriate people. -The team labels planned for deprecation are: - -- ~Configure -- ~Create -- ~Defend -- ~Distribution -- ~Ecosystem -- ~Geo -- ~Gitaly -- ~Growth -- ~Manage -- ~Memory -- ~Monitor -- ~Plan -- ~Release -- ~Secure -- ~Verify - -The following team labels are **true** teams per our [organization structure](https://about.gitlab.com/company/team/structure/#organizational-structure) which will remain post deprecation. +The current team labels are: - ~Delivery - ~Documentation - -The descriptions on the [labels page](https://gitlab.com/gitlab-org/gitlab-ce/-/labels) explain what falls under the -responsibility of each team. - -Within those team labels, we also have the ~backend and ~frontend labels to -indicate if an issue needs backend work, frontend work, or both. +- ~Quality Team labels are always capitalized so that they show up as the first label for any issue. - ## Stage labels Stage labels specify which [DevOps stage][devops-stages] the issue belongs to. -The current stage labels are: - -- ~"devops::manage" -- ~"devops::plan" -- ~"devops::create" -- ~"devops::verify" -- ~"devops::package" -- ~"devops::release" -- ~"devops::configure" -- ~"devops::monitor" -- ~"devops::secure" -- ~"devops::defend" -- ~"devops::growth" -- ~"devops::enablement" +The current stage labels can be found by [searching the labels list for `devops::`](https://gitlab.com/groups/gitlab-org/-/labels?search=devops%3A%3A). These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) and thus are mutually exclusive. -They differ from the [Team labels](#team-labels) because teams may work on -issues outside their stage. - -Normally there is a 1:1 relationship between Stage labels and Team labels, but -any issue can be picked up by any team, depending on current priorities. -So, an issue labeled ~"devops:create" may be scheduled by the ~Plan team, for -example. In such cases, it's usual to include both team labels so each team can -be aware of the progress. - The Stage labels are used to generate the [direction pages][direction-pages] automatically. [devops-stages]: https://about.gitlab.com/direction/#devops-stages @@ -139,51 +94,21 @@ The Stage labels are used to generate the [direction pages][direction-pages] aut Group labels specify which [groups][structure-groups] the issue belongs to. -The current group labels are: - -* ~"group::access" -* ~"group::measure" -* ~"group::source code" -* ~"group::knowledge" -* ~"group::editor" -* ~"group::gitaly" -* ~"group::gitter" -* ~"group::team planning" -* ~"group::enterprise planning" -* ~"group::certify" -* ~"group::ci and runner" -* ~"group::testing" -* ~"group::package" -* ~"group::progressive delivery" -* ~"group::release management" -* ~"group::autodevops and kubernetes" -* ~"group::serverless and paas" -* ~"group::apm" -* ~"group::health" -* ~"group::static analysis" -* ~"group::dynamic analysis" -* ~"group::software composition analysis" -* ~"group::runtime application security" -* ~"group::threat management" -* ~"group::application infrastructure security" -* ~"group::activation" -* ~"group::adoption" -* ~"group::upsell" -* ~"group::retention" -* ~"group::fulfillment" -* ~"group::telemetry" -* ~"group::distribution" -* ~"group::geo" -* ~"group::memory" -* ~"group::ecosystem" - +The current group labels can be found by [searching the labels list for `group::`](https://gitlab.com/groups/gitlab-org/-/labels?search=group%3A%3A). These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) and thus are mutually exclusive. -Groups are nested beneath a particular stage, so only one stage label and one group label -can be applied to a single issue. You can find the groups listed in the -[Product Categories pages][product-categories]. +You can find the groups listed in the [Product Stages, Groups, and Categories][product-categories] page. + +We use the term group to map down product requirements from our product stages. +As a team needs some way to collect the work their members are planning to be assigned to, we use the `~group::` labels to do so. + +Normally there is a 1:1 relationship between Stage labels and Group labels. In the spirit of "Everyone can contribute", +any issue can be picked up by any group, depending on current priorities. For example, an issue labeled ~"devops::create" may be picked up by the ~"group::access" group. + +We also use stage and group labels to help quantify our [throughput](https://about.gitlab.com/handbook/engineering/management/throughput). +Please read [Stage and Group labels in Throughtput](https://about.gitlab.com/handbook/engineering/management/throughput/#stage-and-group-labels-in-throughput) for more information on how the labels are used in this context. [structure-groups]: https://about.gitlab.com/company/team/structure/#groups [product-categories]: https://about.gitlab.com/handbook/product/categories/ @@ -192,15 +117,15 @@ can be applied to a single issue. You can find the groups listed in the The current department labels are: -* ~UX -* ~Quality +- ~UX +- ~Quality ## Specialization labels These labels narrow the [specialization](https://about.gitlab.com/company/team/structure/#specialist) on a unit of work. -* ~frontend -* ~backend +- ~frontend +- ~backend ## Release Scoping labels @@ -248,10 +173,10 @@ There can be multiple facets of the impact. The below is a guideline. If a bug seems to fall between two severity labels, assign it to the higher-severity label. - Example(s) of ~S1 - - Data corruption/loss. + - Data corruption/loss. - Security breach. - - Unable to create an issue or merge request. - - Unable to add a comment or discussion to the issue or merge request. + - Unable to create an issue or merge request. + - Unable to add a comment or thread to the issue or merge request. - Example(s) of ~S2 - Cannot submit changes through the web IDE but the commandline works. - A status widget on the merge request page is not working but information can be seen in the test pipeline page. @@ -261,7 +186,7 @@ If a bug seems to fall between two severity labels, assign it to the higher-seve - Example(s) of ~S4 - Label colors are incorrect. - UI elements are not fully aligned. - + ## Label for community contributors Issues that are beneficial to our users, 'nice to haves', that we currently do @@ -296,7 +221,7 @@ know how difficult the issue is. Additionally: as suitable for people that have never contributed to GitLab before on the [Up For Grabs campaign](http://up-for-grabs.net) - We encourage people that have never contributed to any open source project to - look for [`Accepting merge requests` issues with a weight of 1][firt-timers] + look for [`Accepting merge requests` issues with a weight of 1][first-timers] If you've decided that you would like to work on an issue, please @-mention the [appropriate product manager](https://about.gitlab.com/handbook/product/#who-to-talk-to-for-what) @@ -309,8 +234,8 @@ GitLab team members who apply the ~"Accepting merge requests" label to an issue should update the issue description with a responsible product manager, inviting any potential community contributor to @-mention per above. -[up-for-grabs]: https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=0&sort=weight -[firt-timers]: https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=0&sort=weight&weight=1 +[up-for-grabs]: https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight +[first-timers]: https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight&weight=1 ## Issue triaging diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 3f61ad7cb13..4e9c5c81379 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -74,10 +74,10 @@ request is as follows: can be found by running `grep css-class ./app -R`. 1. Be prepared to answer questions and incorporate feedback into your MR with new commits. Once you have fully addressed a suggestion from a reviewer, click the - "Resolve discussion" button beneath it to mark it resolved. - 1. The merge request author resolves only the discussions they have fully addressed. - If there's an open reply or discussion, a suggestion, a question, or anything else, - the discussion should be left to be resolved by the reviewer. + "Resolve thread" button beneath it to mark it resolved. + 1. The merge request author resolves only the threads they have fully addressed. + If there's an open reply or thread, a suggestion, a question, or anything else, + the thread should be left to be resolved by the reviewer. 1. If your MR touches code that executes shell commands, reads or opens files, or handles paths to files on disk, make sure it adheres to the [shell command guidelines](../shell_commands.md) @@ -103,7 +103,8 @@ If you would like quick feedback on your merge request feel free to mention some from the [core team](https://about.gitlab.com/community/core-team/) or one of the [merge request coaches](https://about.gitlab.com/team/). When having your code reviewed and when reviewing merge requests, please keep the [code review guidelines](../code_review.md) -in mind. +in mind. And if your code also makes changes to the database, or does expensive queries, +check the [database review guidelines](../database_review.md). ### Keep it simple @@ -142,6 +143,37 @@ If the guidelines are not met, the MR will not pass the [Danger checks](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/danger/commit_messages/Dangerfile). For more information see [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/). +Example commit message template that can be used on your machine that embodies the above (guide for [how to apply template](https://codeinthehole.com/tips/a-useful-template-for-commit-messages/)): + +```text +# (If applied, this commit will...) <subject> (Max 50 char) +# |<---- Using a Maximum Of 50 Characters ---->| + + +# Explain why this change is being made +# |<---- Try To Limit Each Line to a Maximum Of 72 Characters ---->| + +# Provide links or keys to any relevant tickets, articles or other resources +# Use issues and merge requests' full URLs instead of short references, +# as they are displayed as plain text outside of GitLab + +# --- COMMIT END --- +# -------------------- +# Remember to +# Capitalize the subject line +# Use the imperative mood in the subject line +# Do not end the subject line with a period +# Subject must contain at least 3 words +# Separate subject from body with a blank line +# Commits that change 30 or more lines across at least 3 files must +# describe these changes in the commit body +# Do not use Emojis +# Use the body to explain what and why vs. how +# Can use multiple lines with "-" for bullet points in body +# For more information: https://chris.beams.io/posts/git-commit/ +# -------------------- +``` + ## Contribution acceptance criteria To make sure that your merge request can be approved, please ensure that it meets diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 5c6ea1f469d..7832850a9f0 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -23,6 +23,7 @@ 1. Code should be written in [US English][us-english] 1. [Go](../go_guide/index.md) 1. [Python](../python_guide/index.md) +1. [Shell scripting](../shell_scripting_guide/index.md) This is also the style used by linting tools such as [RuboCop](https://github.com/rubocop-hq/rubocop) and [Hound CI](https://houndci.com). diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 0311eda1ff1..6c9fa983c96 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -9,31 +9,31 @@ An easy first step is to search for your error in Slack or google "GitLab (my er Available `RAILS_ENV` - - `production` (generally not for your main GDK db, but you may need this for e.g. omnibus) - - `development` (this is your main GDK db) - - `test` (used for tests like rspec) +- `production` (generally not for your main GDK db, but you may need this for e.g. Omnibus) +- `development` (this is your main GDK db) +- `test` (used for tests like rspec) ## Nuke everything and start over If you just want to delete everything and start over with an empty DB (~1 minute): - - `bundle exec rake db:reset RAILS_ENV=development` +- `bundle exec rake db:reset RAILS_ENV=development` If you just want to delete everything and start over with dummy data (~40 minutes). This also does `db:reset` and runs DB-specific migrations: - - `bundle exec rake dev:setup RAILS_ENV=development` +- `bundle exec rake dev:setup RAILS_ENV=development` If your test DB is giving you problems, it is safe to nuke it because it doesn't contain important data: - - `bundle exec rake db:reset RAILS_ENV=test` +- `bundle exec rake db:reset RAILS_ENV=test` ## Migration wrangling - - `bundle exec rake db:migrate RAILS_ENV=development`: Execute any pending migrations that you may have picked up from a MR - - `bundle exec rake db:migrate:status RAILS_ENV=development`: Check if all migrations are `up` or `down` - - `bundle exec rake db:migrate:down VERSION=20170926203418 RAILS_ENV=development`: Tear down a migration - - `bundle exec rake db:migrate:up VERSION=20170926203418 RAILS_ENV=development`: Set up a migration - - `bundle exec rake db:migrate:redo VERSION=20170926203418 RAILS_ENV=development`: Re-run a specific migration +- `bundle exec rake db:migrate RAILS_ENV=development`: Execute any pending migrations that you may have picked up from a MR +- `bundle exec rake db:migrate:status RAILS_ENV=development`: Check if all migrations are `up` or `down` +- `bundle exec rake db:migrate:down VERSION=20170926203418 RAILS_ENV=development`: Tear down a migration +- `bundle exec rake db:migrate:up VERSION=20170926203418 RAILS_ENV=development`: Set up a migration +- `bundle exec rake db:migrate:redo VERSION=20170926203418 RAILS_ENV=development`: Re-run a specific migration ## Manually access the database @@ -45,12 +45,12 @@ bundle exec rails dbconsole RAILS_ENV=development bundle exec rails db RAILS_ENV=development ``` - - `\q`: Quit/exit - - `\dt`: List all tables - - `\d+ issues`: List columns for `issues` table - - `CREATE TABLE board_labels();`: Create a table called `board_labels` - - `SELECT * FROM schema_migrations WHERE version = '20170926203418';`: Check if a migration was run - - `DELETE FROM schema_migrations WHERE version = '20170926203418';`: Manually remove a migration +- `\q`: Quit/exit +- `\dt`: List all tables +- `\d+ issues`: List columns for `issues` table +- `CREATE TABLE board_labels();`: Create a table called `board_labels` +- `SELECT * FROM schema_migrations WHERE version = '20170926203418';`: Check if a migration was run +- `DELETE FROM schema_migrations WHERE version = '20170926203418';`: Manually remove a migration ## FAQ diff --git a/doc/development/database_review.md b/doc/development/database_review.md new file mode 100644 index 00000000000..367a481ee11 --- /dev/null +++ b/doc/development/database_review.md @@ -0,0 +1,134 @@ +# Database Review Guidelines + +This page is specific to database reviews. Please refer to our +[code review guide](code_review.md) for broader advice and best +practices for code review in general. + +## General process + +A database review is required for: + +- Changes that touch the database schema or perform data migrations, + including files in: + - `db/` + - `lib/gitlab/background_migration/` +- Changes to the database tooling, e.g.: + - migration or ActiveRecord helpers in `lib/gitlab/database/` + - load balancing +- Changes that produce SQL queries that are beyond the obvious. It is + generally up to the author of a merge request to decide whether or + not complex queries are being introduced and if they require a + database review. + +A database reviewer is expected to look out for obviously complex +queries in the change and review those closer. If the author does not +point out specific queries for review and there are no obviously +complex queries, it is enough to concentrate on reviewing the +migration only. + +It is preferable to review queries in SQL form and generally accepted +to ask the author to translate any ActiveRecord queries in SQL form +for review. + +### Roles and process + +A Merge Request author's role is to: + +- Decide whether a database review is needed. +- If database review is needed, add the ~database label. +- Use the [database changes](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab/merge_request_templates/Database%20changes.md) + merge request template, or include the appropriate items in the MR description. + +A database **reviewer**'s role is to: + +- Perform a first-pass review on the MR and suggest improvements to the author. +- Once satisfied, relabel the MR with ~"database::reviewed", approve it, and + reassign MR to the database **maintainer** suggested by Reviewer + Roulette. + +A database **maintainer**'s role is to: + +- Perform the final database review on the MR. +- Discuss further improvements or other relevant changes with the + database reviewer and the MR author. +- Finally approve the MR and relabel the MR with ~"database::approved" +- Merge the MR if no other approvals are pending or pass it on to + other maintainers as required (frontend, backend, docs). + +### Distributing review workload + +Review workload is distributed using [reviewer roulette](code_review.md#reviewer-roulette) +([example](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25181#note_147551725)). +The MR author should then co-assign the suggested database +**reviewer**. When they give their sign-off, they will hand over to +the suggested database **maintainer**. + +If reviewer roulette didn't suggest a database reviewer & maintainer, +make sure you have applied the ~database label and rerun the +`danger-review` CI job, or pick someone from the +[`@gl-database` team](https://gitlab.com/groups/gl-database/-/group_members). + +### How to prepare for speedy database reviews + +In order to make reviewing easier and therefore faster, please consider preparing a comment +and details for a database reviewer: + +- Provide queries in SQL form rather than ActiveRecord. +- Format any queries with a SQL query formatter, for example with [sqlformat.darold.net](http://sqlformat.darold.net). +- Consider providing query plans via a link to [explain.depesz.com](https://explain.depesz.com) or another tool instead of textual form. +- For query changes, it is best to provide the SQL query along with a plan *before* and *after* the change. This helps to spot differences quickly. +- When providing query plans, make sure to use good parameter values, so that the query executed is a good example and also hits enough data. Usually, the `gitlab-org` namespace (`namespace_id = 9970`) and the `gitlab-org/gitlab-ce` project (`project_id = 13083`) provides enough data to serve as a good example. + +### How to review for database + +- Check migrations + - Review relational modeling and design choices + - Review migrations follow [database migration style guide](migration_style_guide.md), + for example + - [Check ordering of columns](ordering_table_columns.md) + - [Check indexes are present for foreign keys](migration_style_guide.md#adding-foreign-key-constraints) + - Ensure that migrations execute in a transaction or only contain + concurrent index/foreign key helpers (with transactions disabled) + - Check consistency with `db/schema.rb` and that migrations are [reversible](migration_style_guide.md#reversibility) + - Check queries timing (If any): Queries executed in a migration + need to fit comfortably within `15s` - preferably much less than that - on GitLab.com. +- Check [background migrations](background_migrations.md): + - For data migrations, establish a time estimate for execution + - They should only be used when migrating data in larger tables. + - If a single `update` is below than `1s` the query can be placed + directly in a regular migration (inside `db/migrate`). + - Review queries (for example, make sure batch sizes are fine) + - Establish a time estimate for execution + - Because execution time can be longer than for a regular migration, + it's suggested to treat background migrations as post migrations: + place them in `db/post_migrate` instead of `db/migrate`. Keep in mind + that post migrations are executed post-deployment in production. +- Check [timing guidelines for migrations](#timing-guidelines-for-migrations) +- Query performance + - Check for any obviously complex queries and queries the author specifically + points out for review (if any) + - If not present yet, ask the author to provide SQL queries and query plans + (e.g. by using [chatops](understanding_explain_plans.md#chatops) or direct + database access) + - For given queries, review parameters regarding data distribution + - [Check query plans](understanding_explain_plans.md) and suggest improvements + to queries (changing the query, schema or adding indexes and similar) + - General guideline is for queries to come in below 100ms execution time + - If queries rely on prior migrations that are not present yet on production + (eg indexes, columns), you can use a [one-off instance from the restore + pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd) + in order to establish a proper testing environment. + +### Timing guidelines for migrations + +In general, migrations for a single deploy shouldn't take longer than +1 hour for GitLab.com. The following guidelines are not hard rules, they were +estimated to keep migration timing to a minimum. + +NOTE: **Note:** Keep in mind that all runtimes should be measured against GitLab.com. + +| Migration Type | Execution Time Recommended | Notes | +|----|----|---| +| Regular migrations on `db/migrate` | `3 minutes` | A valid exception are index creation as this can take a long time. | +| Post migrations on `db/post_migrate` | `10 minutes` | | +| Background migrations | --- | Since these are suitable for larger tables, it's not possible to set a precise timing guideline, however, any query must stay well below `10s` of execution time. | diff --git a/doc/development/diffs.md b/doc/development/diffs.md index 5655398c886..ac0b8555360 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -133,4 +133,4 @@ File diff will be suppressed (technically different from collapsed, but behaves Diff Viewers, which can be found on `models/diff_viewer/*` are classes used to map metadata about each type of Diff File. It has information whether it's a binary, which partial should be used to render it or which File extensions this class accounts for. -`DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type in order to check if it can be rendered.
\ No newline at end of file +`DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type in order to check if it can be rendered. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index bfce7488a8d..4776c8348d4 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -179,4 +179,3 @@ By default, the Jaeger search UI is available at <http://localhost:16686/search> TIP: **Tip:** Don't forget that you will need to generate traces by using the application before they appear in the Jaeger UI. - diff --git a/doc/development/documentation/feature-change-workflow.md b/doc/development/documentation/feature-change-workflow.md index ca29353ecbe..00c76fe0f1b 100644 --- a/doc/development/documentation/feature-change-workflow.md +++ b/doc/development/documentation/feature-change-workflow.md @@ -69,7 +69,7 @@ To follow a consistent workflow every month, documentation changes involve the Product Managers, the developer who shipped the feature, and the technical writer for the DevOps stage. Each role is described below. -The Documentation items in the GitLab CE/EE [Feature Proposal issue template](https://gitlab.com/gitlab-org/gitlab-ce/raw/template-improvements-for-documentation/.gitlab/issue_templates/Feature%20proposal.md) +The Documentation items in the GitLab CE/EE [Feature Proposal issue template](https://gitlab.com/gitlab-org/gitlab-ce/raw/master/.gitlab/issue_templates/Feature%20proposal.md) and default merge request template will assist you with following this process. ### Product Manager role @@ -121,27 +121,27 @@ All reviewers can help ensure accuracy, clarity, completeness, and adherence to - **Prior to merging**, documentation changes committed by the developer must be reviewed by: - 1. **The code reviewer** for the MR, to confirm accuracy, clarity, and completeness. - 1. Optionally: Others involved in the work, such as other devs or the PM. - 1. Optionally: The technical writer for the DevOps stage. If not prior to merging, the technical writer will review after the merge. - This helps us ensure that the developer has time to merge good content by the freeze, and that it can be further refined by the release, if needed. - - To decide whether to request this review before the merge, consider the amount of time left before the code freeze, the size of the change, - and your degree of confidence in having users of an RC use your docs as written. - - Pre-merge tech writer reviews should be most common when the code is complete well in advance of the freeze and/or for larger documentation changes. - - You can request a review and if there is not sufficient time to complete it prior to the freeze, - the maintainer can merge the current doc changes (if complete) and create a follow-up doc review issue. - - The technical writer can also help decide what docs to merge before the freeze and whether to work on further changes in a follow up MR. - - **To request a pre-merge technical writer review**, assign the writer listed for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). - - **To request a post-merge technical writer review**, [create an issue for one using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review) and link it from the MR that makes the doc change. - 1. **The maintainer** who is assigned to merge the MR, to verify clarity, completeness, and quality, to the best of their ability. + 1. **The code reviewer** for the MR, to confirm accuracy, clarity, and completeness. + 1. Optionally: Others involved in the work, such as other devs or the PM. + 1. Optionally: The technical writer for the DevOps stage. If not prior to merging, the technical writer will review after the merge. + This helps us ensure that the developer has time to merge good content by the freeze, and that it can be further refined by the release, if needed. + - To decide whether to request this review before the merge, consider the amount of time left before the code freeze, the size of the change, + and your degree of confidence in having users of an RC use your docs as written. + - Pre-merge tech writer reviews should be most common when the code is complete well in advance of the freeze and/or for larger documentation changes. + - You can request a review and if there is not sufficient time to complete it prior to the freeze, + the maintainer can merge the current doc changes (if complete) and create a follow-up doc review issue. + - The technical writer can also help decide what docs to merge before the freeze and whether to work on further changes in a follow up MR. + - **To request a pre-merge technical writer review**, assign the writer listed for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). + - **To request a post-merge technical writer review**, [create an issue for one using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review) and link it from the MR that makes the doc change. + 1. **The maintainer** who is assigned to merge the MR, to verify clarity, completeness, and quality, to the best of their ability. - Upon merging, if a technical writer review has not been performed and there is not yet a linked issue for a follow-up review, the maintainer should [create an issue using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review), link it from the MR, and mention the original MR author in the new issue. Alternatively, the maintainer can ask the MR author to create and link this issue before the MR is merged. - After merging, documentation changes are reviewed by: - 1. The technical writer--**if** their review was not performed prior to the merge. - 2. Optionally: by the PM (for accuracy and to ensure it's consistent with the vision for how the product will be used). + 1. The technical writer -- **if** their review was not performed prior to the merge. + 1. Optionally: by the PM (for accuracy and to ensure it's consistent with the vision for how the product will be used). Any party can raise the item to the PM for review at any point: the dev, the technical writer, or the PM, who can request/plan a review at the outset. ### Technical Writer role diff --git a/doc/development/documentation/improvement-workflow.md b/doc/development/documentation/improvement-workflow.md index a12c3d5ea7b..80fbd4b6427 100644 --- a/doc/development/documentation/improvement-workflow.md +++ b/doc/development/documentation/improvement-workflow.md @@ -52,9 +52,9 @@ To request a post-merge review, [create an issue for one using the Doc Review te **3. Maintainer** 1. Review by assigned maintainer, who can always request/require the above reviews. Maintainer review can occur before or after a technical writer review. -2. Ensure a release milestone of the format XX.Y is set. If the freeze for that release has begun, add the label `pick into <XX.Y>` unless this change is not required for the release. In that case, simply change the milestone. -3. If EE and CE MRs exist, merge the EE MR first, then the CE MR. -4. After merging, if there has not been a technical writer review and an issue for a follow-up review was not already created and linked from the MR, [create the issue using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review) and link it from the MR. +1. Ensure a release milestone of the format XX.Y is set. If the freeze for that release has begun, add the label `pick into <XX.Y>` unless this change is not required for the release. In that case, simply change the milestone. +1. If EE and CE MRs exist, merge the EE MR first, then the CE MR. +1. After merging, if there has not been a technical writer review and an issue for a follow-up review was not already created and linked from the MR, [create the issue using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review) and link it from the MR. ## Other ways to help diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index cbdc0a3a174..c9ae00d148a 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -43,7 +43,7 @@ Meanwhile, anyone can contribute [documentation improvements](improvement-workfl ## Markdown and styles -[GitLab docs](https://gitlab.com/gitlab-com/gitlab-docs) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown) +[GitLab docs](https://gitlab.com/gitlab-org/gitlab-docs) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown) as its markdown rendering engine. See the [GitLab Markdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) for a complete Kramdown reference. Adhere to the [Documentation Style Guide](styleguide.md). If a style standard is missing, you are welcome to suggest one via a merge request. @@ -57,35 +57,22 @@ See the [Structure](styleguide.md#structure) section of the [Documentation Style We currently maintain two sets of docs: one in the [gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc) repo and one in [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc). -They are similar, and most pages are identical, but they are different repositories. -With the single codebase effort, we want to make those two sets identical, so when the -time comes to have only one codebase, we'll be ready. - -Here are some links to get you up to speed with the current effort: - -- [CE/EE codebases blueprint](https://about.gitlab.com/handbook/engineering/infrastructure/blueprint/ce-ee-codebases/) -- [CE/EE codebases merge design](https://about.gitlab.com/handbook/engineering/infrastructure/design/merge-ce-ee-codebases/) -- [Single docs codebase epic](https://gitlab.com/groups/gitlab-org/-/epics/199) -- [Issue board of related issues](https://gitlab.com/groups/gitlab-org/-/boards/981090?&label_name[]=Documentation&label_name[]=single%20codebase) -- [Related merge requests](https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&label_name[]=Documentation&label_name[]=single%20codebase) -- [Visualize the existing diffs](https://leipert-projects.gitlab.io/is-gitlab-pretty-yet/diff/?search=%5Edoc) +They are identical, but they are different repositories. When the +time comes to have only one codebase for the GitLab project, we'll be ready. ### CE first -After a given documentation path is aligned across CE and EE, all merge requests -affecting that path must be submitted to CE, regardless of the content it has. -This means that: +All merge requests for documentation must be submitted to CE, regardless of the content +it has. This means that: -- For **EE-only docs changes**, you only have to submit a CE MR. +- For **EE-only docs changes**, you only have to submit an MR in the CE project. - For **EE-only features** that touch both the code and the docs, you have to submit - an EE MR containing all changes, and a CE MR containing only the docs changes + an EE MR containing all code changes, and a CE MR containing only the docs changes and without a changelog entry. This might seem like a duplicate effort, but it's only for the short term. -A list of the already aligned docs can be found in -[the epic description](https://gitlab.com/groups/gitlab-org/-/epics/199#ee-specific-lines-check). -Since the docs will be combined, it's crucial to add the relevant +Since the CE and EE docs are combined, it's crucial to add the relevant [product badges](styleguide.md#product-badges) for all EE documentation, so that we can discern which features belong to which tier. @@ -94,19 +81,9 @@ we can discern which features belong to which tier. There's a special test in place ([`ee_specific_check.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/ee_specific_check/ee_specific_check.rb)), which, among others, checks and prevents creating/editing new files and directories -in EE under `doc/`. - -We have a long list of documentation paths that are either whitelisted or not. -Paths in the whitelist (not commented out) will not be subject to the test, -which means you are allowed to create/change docs content in EE for the time -being. The goal is to not have any doc whitelisted. - -At the time of this writing, the only items left to be aligned are the API docs: - -- `doc/api/*` ([issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/60045) / [merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27491)) - -Eventually, once all docs are aligned, we'll remove any doc reference from that -script, so it catches everything. +in EE under `doc/`. This should fail when changes to anything in `/doc` are submitted +in an EE MR. To pass the test, simply remove the docs changes from the EE MR, and +[submit them in CE](#ce-first). ## Changing document location @@ -134,18 +111,18 @@ For example, if you were to move `doc/workflow/lfs/lfs_administration.md` to 1. Copy `doc/workflow/lfs/lfs_administration.md` to `doc/administration/lfs.md` 1. Replace the contents of `doc/workflow/lfs/lfs_administration.md` with: - ```md - This document was moved to [another location](../../administration/lfs.md). - ``` + ```md + This document was moved to [another location](../../administration/lfs.md). + ``` 1. Find and replace any occurrences of the old location with the new one. A quick way to find them is to use `git grep`. First go to the root directory where you cloned the `gitlab-ce` repository and then do: - ```sh - git grep -n "workflow/lfs/lfs_administration" - git grep -n "lfs/lfs_administration" - ``` + ```sh + git grep -n "workflow/lfs/lfs_administration" + git grep -n "lfs/lfs_administration" + ``` NOTE: **Note:** If the document being moved has any Disqus comments on it, there are extra steps @@ -193,7 +170,7 @@ Disqus uses an identifier per page, and for docs.gitlab.com, the page identifier is configured to be the page URL. Therefore, when we change the document location, we need to preserve the old URL as the same Disqus identifier. -To do that, add to the frontmatter the variable `redirect_from`, +To do that, add to the frontmatter the variable `disqus_identifier`, using the old URL as value. For example, let's say I moved the document available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, `https://docs.gitlab.com/my-new-location/index.html`. @@ -202,11 +179,11 @@ Into the **new document** frontmatter add the following: ```yaml --- -redirect_from: 'https://docs.gitlab.com/my-old-location/README.html' +disqus_identifier: 'https://docs.gitlab.com/my-old-location/README.html' --- ``` -Note: it is necessary to include the file name in the `redirect_from` URL, +Note: it is necessary to include the file name in the `disqus_identifier` URL, even if it's `index.html` or `README.html`. ## Branch naming @@ -319,45 +296,45 @@ You can combine one or more of the following: 1. **Linking to an anchor link.** Use `anchor` as part of the `help_page_path` method: - ```haml - = link_to 'Help page', help_page_path('user/permissions', anchor: 'anchor-link') - ``` + ```haml + = link_to 'Help page', help_page_path('user/permissions', anchor: 'anchor-link') + ``` 1. **Opening links in a new tab.** This should be the default behavior: - ```haml - = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' - ``` + ```haml + = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' + ``` 1. **Linking to a circle icon.** Usually used in settings where a long description cannot be used, like near checkboxes. You can basically use any font awesome icon, but prefer the `question-circle`: - ```haml - = link_to icon('question-circle'), help_page_path('user/permissions') - ``` + ```haml + = link_to icon('question-circle'), help_page_path('user/permissions') + ``` 1. **Using a button link.** Useful in places where text would be out of context with the rest of the page layout: - ```haml - = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' - ``` + ```haml + = link_to 'Help page', help_page_path('user/permissions'), class: 'btn btn-info' + ``` 1. **Using links inline of some text.** - ```haml - Description to #{link_to 'Help page', help_page_path('user/permissions')}. - ``` + ```haml + Description to #{link_to 'Help page', help_page_path('user/permissions')}. + ``` 1. **Adding a period at the end of the sentence.** Useful when you don't want the period to be part of the link: - ```haml - = succeed '.' do - Learn more in the - = link_to 'Help page', help_page_path('user/permissions') - ``` + ```haml + = succeed '.' do + Learn more in the + = link_to 'Help page', help_page_path('user/permissions') + ``` ### GitLab `/help` tests @@ -384,7 +361,7 @@ on how the left-side navigation menu is built and updated. NOTE: **Note:** To preview your changes to documentation locally, follow this -[development guide](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md#development-when-contributing-to-gitlab-documentation) or [these instructions for GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/gitlab_docs.md). +[development guide](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md#development-when-contributing-to-gitlab-documentation) or [these instructions for GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/gitlab_docs.md). The live preview is currently enabled for the following projects: @@ -408,7 +385,7 @@ You will need to push a branch to those repositories, it doesn't work for forks. The `review-docs-deploy*` job will: -1. Create a new branch in the [gitlab-docs](https://gitlab.com/gitlab-com/gitlab-docs) +1. Create a new branch in the [gitlab-docs](https://gitlab.com/gitlab-org/gitlab-docs) project named after the scheme: `$DOCS_GITLAB_REPO_SUFFIX-$CI_ENVIRONMENT_SLUG`, where `DOCS_GITLAB_REPO_SUFFIX` is the suffix for each product, e.g, `ce` for CE, etc. @@ -464,7 +441,7 @@ If you want to know the in-depth details, here's what's really happening: 1. The preview URL is shown both at the job output and in the merge request widget. You also get the link to the remote pipeline. 1. In the docs project, the pipeline is created and it - [skips the test jobs](https://gitlab.com/gitlab-com/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) + [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) to lower the build time. 1. Once the docs site is built, the HTML files are uploaded as artifacts. 1. A specific Runner tied only to the docs project, runs the Review App job @@ -488,15 +465,17 @@ Currently, the following tests are in place: that all cURL examples in API docs use the full switches. It's recommended to [check locally](#previewing-the-changes-live) before pushing to GitLab by executing the command `bundle exec nanoc check internal_links` on your local - [`gitlab-docs`](https://gitlab.com/gitlab-com/gitlab-docs) directory. + [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) directory. In addition, + `docs-lint` also runs [markdownlint](styleguide.md#markdown-rules) to ensure the + markdown is consistent across all documentation. 1. [`ee_compat_check`](../automatic_ce_ee_merge.md#avoiding-ce-ee-merge-conflicts-beforehand) (runs on CE only): - When you submit a merge request to GitLab Community Edition (CE), - there is this additional job that runs against Enterprise Edition (EE) - and checks if your changes can apply cleanly to the EE codebase. - If that job fails, read the instructions in the job log for what to do next. - As CE is merged into EE once a day, it's important to avoid merge conflicts. - Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is - essential to avoid them. + When you submit a merge request to GitLab Community Edition (CE), + there is this additional job that runs against Enterprise Edition (EE) + and checks if your changes can apply cleanly to the EE codebase. + If that job fails, read the instructions in the job log for what to do next. + As CE is merged into EE once a day, it's important to avoid merge conflicts. + Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is + essential to avoid them. 1. [`ee-files-location-check`/`ee-specific-lines-check`](#ee-specific-lines-check) (runs on EE only): This test ensures that no new files/directories are created/changed in EE. All docs should be submitted in CE instead, regardless the tier they are on. @@ -559,15 +538,16 @@ A file with `proselint` configuration must be placed in a #### `markdownlint` `markdownlint` checks that certain rules ([example](https://github.com/DavidAnson/markdownlint/blob/master/README.md#rules--aliases)) - are followed for Markdown syntax. - Our [Documentation Style Guide](styleguide.md) and [Markdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) - elaborate on which choices must be made when selecting Markdown syntax for - GitLab documentation. This tool helps catch deviations from those guidelines. +are followed for Markdown syntax. Our [Documentation Style Guide](styleguide.md) and +[Markdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) +elaborate on which choices must be made when selecting Markdown syntax for GitLab +documentation. This tool helps catch deviations from those guidelines, and matches the +tests run on the documentation by [`docs-lint`](#testing). `markdownlint` can be used [on the command line](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli--), - either on a single Markdown file or on all Markdown files in a project. For example, to run - `markdownlint` on all documentation in the [`gitlab-ce` project](https://gitlab.com/gitlab-org/gitlab-ce), - run the following commands from within the `gitlab-ce` project: +either on a single Markdown file or on all Markdown files in a project. For example, to run +`markdownlint` on all documentation in the [`gitlab-ce` project](https://gitlab.com/gitlab-org/gitlab-ce), +run the following commands from within the `gitlab-ce` project: ```sh cd doc @@ -597,7 +577,7 @@ The following sample `markdownlint` configuration modifies the available default "line-length": false, "no-trailing-punctuation": false, "ol-prefix": { "style": "one" }, - "blanks-around-fences": false, + "blanks-around-fences": true, "no-inline-html": { "allowed_elements": [ "table", @@ -612,11 +592,15 @@ The following sample `markdownlint` configuration modifies the available default "a", "strong", "i", - "div" + "div", + "b" ] }, "hr-style": { "style": "---" }, - "fenced-code-language": false + "code-block-style": { "style": "fenced" }, + "fenced-code-language": false, + "no-duplicate-header": { "allow_different_nesting": true }, + "commands-show-output": false } ``` diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 20eeebf444f..753a636a779 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -4,9 +4,9 @@ description: "Learn how GitLab docs' global navigation works and how to add new # Global navigation -> - [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/362) +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/362) in GitLab 11.6. -> - [Updated](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/482) in GitLab 12.1. +> - [Updated](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/482) in GitLab 12.1. The global nav adds to the left sidebar the ability to navigate and explore the contents of GitLab's documentation. @@ -25,7 +25,7 @@ To add a new doc to the nav, first and foremost, check with the technical writin Once you get their approval and their guidance in regards to the position on the nav, read trhough this page to understand how it works, and submit a merge request to the docs site, adding the doc you wish to include in the nav into the -[global nav data file](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/content/_data/global-nav.yaml). +[global nav data file](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/global-nav.yaml). Don't forget to ask a technical writer to review your changes before merging. @@ -70,7 +70,7 @@ the data among the nav in containers properly [styled](#css-classes). ### Data file -The [data file](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/content/_data/global-nav.yaml) +The [data file](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/global-nav.yaml) is structured in three components: sections, categories, and docs. #### Sections @@ -248,9 +248,9 @@ Examples: ### Layout file (logic) -The [layout](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/layouts/global_nav.html) +The [layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/global_nav.html) is fed by the [data file](#data-file), builds the global nav, and is rendered by the -[default](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/layouts/default.html) layout. +[default](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/default.html) layout. There are three main considerations on the logic built for the nav: diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 6dd12b5efa7..1aef0ed855c 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -4,14 +4,14 @@ description: "Learn how GitLab's documentation website is architectured." # Documentation site architecture -Learn how we build and architecture [`gitlab-docs`](https://gitlab.com/gitlab-com/gitlab-docs) +Learn how we build and architecture [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) and deploy it to <https://docs.gitlab.com>. ## Repository While the source of the documentation content is stored in GitLab's respective product repositories, the source that is used to build the documentation site _from that content_ -is located at <https://gitlab.com/gitlab-com/gitlab-docs>. +is located at <https://gitlab.com/gitlab-org/gitlab-docs>. The following diagram illustrates the relationship between the repositories from where content is sourced, the `gitlab-docs` project, and the published output. @@ -43,7 +43,7 @@ from where content is sourced, the `gitlab-docs` project, and the published outp G --> L ``` -See the [README there](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md) +See the [README there](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md) for detailed information. ## Assets @@ -76,7 +76,7 @@ read through the [global navigation](global_nav.md) doc. The docs site is deployed to production with GitLab Pages, and previewed in merge requests with Review Apps. -The deployment aspects will be soon transferred from the [original document](https://gitlab.com/gitlab-com/gitlab-docs/blob/master/README.md) +The deployment aspects will be soon transferred from the [original document](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md) to this page. <!-- diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index fe676efa94d..025a946da0e 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -127,7 +127,7 @@ Notes: ## Help and feedback section -The "help and feedback" section (introduced by [!319](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/319)) displayed at the end of each document +The "help and feedback" section (introduced by [!319](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/319)) displayed at the end of each document can be omitted from the doc by adding a key into the its frontmatter: ```yaml @@ -142,7 +142,7 @@ you must check with a technical writer before doing so. ### Disqus We also have integrated the docs site with Disqus (introduced by -[!151](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/151)), +[!151](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/151)), allowing our users to post comments. To omit only the comments from the feedback section, use the following diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index d9cea0614c3..c1e3eb9680b 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -36,17 +36,17 @@ For the Troubleshooting sections, people in GitLab Support can merge additions t Include any media types/sources if the content is relevant to readers. You can freely include or link presentations, diagrams, videos, etc.; no matter who it was originally composed for, if it is helpful to any of our audiences, we can include it. - - If you use an image that has a separate source file (for example, a vector or diagram format), link the image to the source file so that it may be reused or updated by anyone. - - Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. +- If you use an image that has a separate source file (for example, a vector or diagram format), link the image to the source file so that it may be reused or updated by anyone. +- Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. ### No special types -In the software industry, it is a best practice to organize documentatioin in different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): +In the software industry, it is a best practice to organize documentation in different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): 1. Tutorials -2. How-to guides -3. Explanation -4. Reference (for example, a glossary) +1. How-to guides +1. Explanation +1. Reference (for example, a glossary) At GitLab, we have so many product changes in our monthly releases that we can't afford to continually update multiple types of information. If we have multiple types, the information will become outdated. Therefore, we have a [single template](structure.md) for documentation. @@ -107,6 +107,22 @@ Hard-coded HTML is valid, although it's discouraged to be used while we have `/h - Special styling is required. - Reviewed and approved by a technical writer. +### Markdown Rules + +GitLab ensures that the Markdown used across all documentation is consistent, as +well as easy to review and maintain, by testing documentation changes with +[Markdownlint (mdl)](https://github.com/markdownlint/markdownlint). This lint test +checks many common problems with Markdown, and fails when any document has an issue +with Markdown formatting that may cause the page to render incorrectly within GitLab. +It will also fail when a document is using non-standard Markdown (which may render +correctly, but is not the current standard in GitLab documentation). + +Each formatting issue that mdl checks has an associated [rule](https://github.com/markdownlint/markdownlint/blob/master/docs/RULES.md), +and the rules that are currently enabled for GitLab documentation are listed in the +[`.mdlrc.style`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.mdlrc.style) file. +Configuration options are set in the [`.mdlrc`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.mdlrc.style) +file. + ## Structure ### Organize by topic, not by type @@ -142,9 +158,9 @@ The table below shows what kind of documentation goes where. ### Working with directories and files 1. When you create a new directory, always start with an `index.md` file. - Do not use another file name and **do not** create `README.md` files. + Do not use another file name and **do not** create `README.md` files. 1. **Do not** use special characters and spaces, or capital letters in file names, - directory names, branch names, and anything that generates a path. + directory names, branch names, and anything that generates a path. 1. When creating a new document and it has more than one word in its name, make sure to use underscores instead of spaces or dashes (`-`). For example, a proper naming would be `import_projects_from_github.md`. The same rule @@ -221,14 +237,14 @@ Do not include the same information in multiple places. [Link to a SSOT instead. - Use sentence case for titles, headings, labels, menu items, and buttons. - Insert an empty line between different markups (e.g., after every paragraph, header, list, etc). Example: - ```md - ## Header + ```md + ## Header - Paragraph. + Paragraph. - - List item 1 - - List item 2 - ``` + - List item 1 + - List item 2 + ``` ### Tables overlapping the TOC @@ -267,32 +283,52 @@ Check specific punctuation rules for [list items](#list-items) below. ## List items -- Always start list items with a capital letter. +- Always start list items with a capital letter, unless they are parameters or commands + that are in backticks, or similar. - Always leave a blank line before and after a list. -- Begin a line with spaces (not tabs) to denote a subitem. -- To nest subitems, indent them with two spaces. -- To nest code blocks, indent them with four spaces. -- Only use ordered lists when their items describe a sequence of steps to follow. +- Begin a line with spaces (not tabs) to denote a [nested subitem](#nesting-inside-a-list-item). +- Only use ordered lists when their items describe a sequence of steps to follow: + + Do: + + These are the steps to do something: + + 1. First, do step 1 + 1. Then, do step 2 + 1. Finally, do step 3 + + Don't: + + This is a list of different features: + + 1. Feature 1 + 1. Feature 2 + 1. Feature 3 **Markup:** - Use dashes (`-`) for unordered lists instead of asterisks (`*`). -- Use the number one (`1`) for each item in an ordered list. - When rendered, the list items will appear with sequential numbering. +- Prefix `1.` to each item in an ordered list. + When rendered, the list items will appear with sequential numbering automatically. **Punctuation:** -- Do not add commas (`,`) or semicolons (`;`) to the end of a list item. -- Only add periods to the end of a list item if the item consists of a complete sentence. The [definition of full sentence](https://www2.le.ac.uk/offices/ld/resources/writing/grammar/grammar-guides/sentence) is: _"a complete sentence always contains a verb, expresses a complete idea, and makes sense standing alone"_. -- Be consistent throughout the list: if the majority of the items do not end in a period, do not end any of the items in a period, even if they consist of a complete sentence. The opposite is also valid: if the majority of the items end with a period, end all with a period. +- Do not add commas (`,`) or semicolons (`;`) to the end of list items. +- Only add periods to the end of a list item if the item consists of a complete sentence. + The [definition of full sentence](https://www2.le.ac.uk/offices/ld/resources/writing/grammar/grammar-guides/sentence) + is: _"a complete sentence always contains a verb, expresses a complete idea, and makes sense standing alone"_. +- Be consistent throughout the list: if the majority of the items do not end in a period, + do not end any of the items in a period, even if they consist of a complete sentence. + The opposite is also valid: if the majority of the items end with a period, end + all with a period. - Separate list items from explanatory text with a colon (`:`). For example: - ```md - The list is as follows: + ```md + The list is as follows: - - First item: this explains the first item. - - Second item: this explains the second item. - ``` + - First item: this explains the first item. + - Second item: this explains the second item. + ``` **Examples:** @@ -314,12 +350,86 @@ Do: - Let's say this is also a complete sentence. - Not a complete sentence. -Don't: +Don't (third item should have a `.` to match the first and second items): - Let's say this is a complete sentence. - Let's say this is also a complete sentence. - Not a complete sentence +### Nesting inside a list item + +It is possible to nest items under a list item, so that they render with the same indentation +as the list item. This can be done with: + +- [Code blocks](#code-blocks) +- [Blockquotes](#blockquotes) +- [Alert boxes](#alert-boxes) +- [Images](#images) + +Items nested in lists should always align with the first character of the list item. +In unordered lists (using `-`), this means two spaces for each level of indentation: + +~~~md +- Unordered list item 1 + + A line nested using 2 spaces to align with the `U` above. + +- Unordered list item 2 + + > A quote block that will nest + > inside list item 2. + +- Unordered list item 3 + + ```text + a codeblock that will next inside list item 3 + ``` + +- Unordered list item 4 + +  +~~~ + +For ordered lists, use three spaces for each level of indentation: + +~~~md +1. Ordered list item 1 + + A line nested using 3 spaces to align with the `O` above. + +1. Ordered list item 2 + + > A quote block that will nest + > inside list item 2. + +1. Ordered list item 3 + + ```text + a codeblock that will next inside list item 3 + ``` + +1. Ordered list item 4 + +  +~~~ + +You can nest full lists inside other lists using the same rules as above. If you wish +to mix types, that is also possible, as long as you don't mix items at the same level: + +``` +1. Ordered list item one. +1. Ordered list item two. + - Nested unordered list item one. + - Nested unordered list item two. +1. Ordered list item three. + +- Unordered list item one. +- Unordered list item two. + 1. Nested ordered list item one. + 1. Nested ordered list item two. +- Unordered list item three. +``` + ## Quotes Valid for markdown content only, not for frontmatter entries: @@ -340,7 +450,7 @@ For other punctuation rules, please refer to the links shift too, which eventually leads to dead links. If you think it is compelling to add numbers in headings, make sure to at least discuss it with someone in the Merge Request. -- [Avoid using symbols and special chars](https://gitlab.com/gitlab-com/gitlab-docs/issues/84) +- [Avoid using symbols and special chars](https://gitlab.com/gitlab-org/gitlab-docs/issues/84) in headers. Whenever possible, they should be plain and short text. - Avoid adding things that show ephemeral statuses. For example, if a feature is considered beta or experimental, put this info in a note, not in the heading. @@ -488,7 +598,7 @@ You can link any up-to-date video that is useful to the GitLab user. ### Embed videos -> [Introduced](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/472) in GitLab 12.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/472) in GitLab 12.1. GitLab docs (docs.gitlab.com) support embedded videos. @@ -504,16 +614,16 @@ To embed a video, follow the instructions below and make sure you have your MR reviewed and approved by a technical writer. 1. Copy the code below and paste it into your markdown file. - Leave a blank line above and below it. Do NOT edit the code - (don't remove or add any spaces, etc). + Leave a blank line above and below it. Do NOT edit the code + (don't remove or add any spaces, etc). 1. On YouTube, visit the video URL you want to display. Copy - the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) - and replace the video title and link in the line under `<div class="video-fallback">`. + the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) + and replace the video title and link in the line under `<div class="video-fallback">`. 1. On YouTube, click **Share**, then **Embed**. 1. Copy the `<iframe>` source (`src`) **URL only** - (`https://www.youtube.com/embed/VIDEO-ID`), - and paste it, replacing the content of the `src` field in the - `iframe` tag. + (`https://www.youtube.com/embed/VIDEO-ID`), + and paste it, replacing the content of the `src` field in the + `iframe` tag. ```html leave a blank line here @@ -545,15 +655,16 @@ nicely on different mobile devices. ## Code blocks -- Always wrap code added to a sentence in inline code blocks (``` ` ```). +- Always wrap code added to a sentence in inline code blocks (`` ` ``). E.g., `.gitlab-ci.yml`, `git add .`, `CODEOWNERS`, `only: master`. File names, commands, entries, and anything that refers to code should be added to code blocks. To make things easier for the user, always add a full code block for things that can be useful to copy and paste, as they can easily do it with the button on code blocks. +- Add a blank line above and below code blocks. - For regular code blocks, always use a highlighting class corresponding to the language for better readability. Examples: - ````md + ~~~md ```ruby Ruby code ``` @@ -563,16 +674,17 @@ nicely on different mobile devices. ``` ```md - Markdown code + [Markdown code example](example.md) ``` ```text - Code for which no specific highlighting class is available. + Code or text for which no specific highlighting class is available. ``` - ```` + ~~~ -- To display raw markdown instead of rendered markdown, use four backticks on their own lines around the - markdown to display. See [example](https://gitlab.com/gitlab-org/gitlab-ce/blob/8c1991b9bb7e3b8d606481fdea316d633cfa5eb7/doc/development/documentation/styleguide.md#L275-287). +- To display raw markdown instead of rendered markdown, you can use triple backticks + with `md`, like the `Markdown code` example above, unless you want to include triple + backticks in the code block as well. In that case, use triple tildes (`~~~`) instead. - For a complete reference on code blocks, check the [Kramdown guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/#code-blocks). ## Alert boxes @@ -595,7 +707,7 @@ In most cases, content considered for a note should be included: #### When to use Use a note when there is a reason that most or all readers who browse the -section should see the content. That is, if missed, it’s likely to cause +section should see the content. That is, if missed, it’s likely to cause major trouble for a minority of users or significant trouble for a majority of users. @@ -731,24 +843,24 @@ a helpful link back to how the feature was developed. - For features that need to declare the GitLab version that the feature was introduced. Text similar to the following should be added immediately below the heading as a blockquote: - ```md - > Introduced in GitLab 11.3. - ``` + ```md + > Introduced in GitLab 11.3. + ``` - Whenever possible, version text should have a link to the issue, merge request, or epic that introduced the feature. An issue is preferred over a merge request, and a merge request is preferred over an epic. For example: - ```md - > [Introduced](<link-to-issue>) in GitLab 11.3. - ``` + ```md + > [Introduced](<link-to-issue>) in GitLab 11.3. + ``` - If the feature is only available in GitLab Enterprise Edition, mention the [paid tier](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers) the feature is available in: - ```md - > [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. - ``` + ```md + > [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. + ``` ### Removing version text @@ -767,10 +879,10 @@ Other text includes deprecation notices and version-specific how-to information. When a feature is available in EE-only tiers, add the corresponding tier according to the feature availability: +- For GitLab Core and GitLab.com Free: `**(CORE)**`. - For GitLab Starter and GitLab.com Bronze: `**(STARTER)**`. - For GitLab Premium and GitLab.com Silver: `**(PREMIUM)**`. - For GitLab Ultimate and GitLab.com Gold: `**(ULTIMATE)**`. -- For GitLab Core and GitLab.com Free: `**(CORE)**`. To exclude GitLab.com tiers (when the feature is not available in GitLab.com), add the keyword "only": @@ -782,6 +894,7 @@ keyword "only": For GitLab.com only tiers (when the feature is not available for self-hosted instances): +- For GitLab Free and higher tiers: `**(FREE ONLY)**`. - For GitLab Bronze and higher tiers: `**(BRONZE ONLY)**`. - For GitLab Silver and higher tiers: `**(SILVER ONLY)**`. - For GitLab Gold: `**(GOLD ONLY)**`. @@ -801,7 +914,7 @@ GitLab.com Free, and all higher tiers. ### How it works -Introduced by [!244](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/244), +Introduced by [!244](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/244), the special markup `**(STARTER)**` will generate a `span` element to trigger the badges and tooltips (`<span class="badge-trigger starter">`). When the keyword "only" is added, the corresponding GitLab.com badge will not be displayed. @@ -855,14 +968,14 @@ When there is a list of steps to perform, usually that entails editing the configuration file and reconfiguring/restarting GitLab. In such case, follow the style below as a guide: -```md +````md **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - external_url "https://gitlab.example.com" - ``` + ```ruby + external_url "https://gitlab.example.com" + ``` 1. Save the file and [reconfigure] GitLab for the changes to take effect. @@ -872,17 +985,16 @@ the style below as a guide: 1. Edit `config/gitlab.yml`: - ```yaml - gitlab: - host: "gitlab.example.com" - ``` + ```yaml + gitlab: + host: "gitlab.example.com" + ``` 1. Save the file and [restart] GitLab for the changes to take effect. - [reconfigure]: path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure [restart]: path/to/administration/restart_gitlab.md#installations-from-source -``` +```` In this case: @@ -901,9 +1013,9 @@ on this document. Further explanation is given below. - Every method must have the REST API request. For example: - ``` - GET /projects/:id/repository/branches - ``` + ``` + GET /projects/:id/repository/branches + ``` - Every method must have a detailed [description of the parameters](#method-description). @@ -914,7 +1026,7 @@ on this document. Further explanation is given below. The following can be used as a template to get started: -````md +~~~md ## Descriptive title One or two sentence description of what endpoint does. @@ -942,7 +1054,7 @@ Example response: } ] ``` -```` +~~~ ### Fake tokens @@ -955,7 +1067,7 @@ You can use the following fake tokens as examples. | Token type | Token value | |:----------------------|:-------------------------------------------------------------------| -| Private user token | `<your_access_token>` | +| Private user token | `<your_access_token>` | | Personal access token | `n671WNGecHugsdEDPsyo` | | Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | | Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | @@ -970,7 +1082,7 @@ You can use the following fake tokens as examples. ### Method description Use the following table headers to describe the methods. Attributes should -always be in code blocks using backticks (``` ` ```). +always be in code blocks using backticks (`` ` ``). ```md | Attribute | Type | Required | Description | @@ -1076,6 +1188,6 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "domain_ [cURL]: http://curl.haxx.se/ "cURL website" [single spaces]: http://www.slate.com/articles/technology/technology/2011/01/space_invaders.html -[gfm]: https://docs.gitlab.com/ce/user/markdown.html#newlines "GitLab flavored markdown documentation" +[gfm]: ../../user/markdown.md#newlines "GitLab flavored markdown documentation" [ce-1242]: https://gitlab.com/gitlab-org/gitlab-ce/issues/1242 [doc-restart]: ../../administration/restart_gitlab.md "GitLab restart documentation" diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 0abfe4b82a4..9f488fac7d0 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -6,5 +6,5 @@ description: Learn the processes for contributing to GitLab's documentation. Documentation workflows at GitLab differ depending on the reason for the change: -- [Documentation process for feature changes](feature-change-workflow.md) - The documentation is being created or updated as part of the development and release of a new or enhanced feature. This process involves the developer of the feature (who includes new/updated documentation files as part of the same merge request containing the feature's code) and also involves the product manager and technical writer who are listed for the feature's [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). +- [Documentation process for feature changes](feature-change-workflow.md) - The documentation is being created or updated as part of the development and release of a new or enhanced feature. This process involves the developer of the feature (who includes new/updated documentation files as part of the same merge request containing the feature's code) and also involves the product manager and technical writer who are listed for the feature's [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). - [Documentation improvement workflow](improvement-workflow.md) - All documentation additions not associated with a feature release. Documentation is being created or updated to improve accuracy, completeness, ease of use, or any reason other than a feature change. Anyone (and everyone) can contribute a merge request for this type of change at any time. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 7131b717353..1358851f3cd 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -125,20 +125,24 @@ This also applies to views. ### EE features based on CE features For features that build on existing CE features, write a module in the `EE` -namespace and `prepend` it in the CE class, on the last line of the file that -the class resides in. This makes conflicts less likely to happen during CE to EE -merges because only one line is added to the CE class - the `prepend` line. For -example, to prepend a module into the `User` class you would use the following -approach: +namespace and inject it in the CE class, on the last line of the file that the +class resides in. This makes conflicts less likely to happen during CE to EE +merges because only one line is added to the CE class - the line that injects +the module. For example, to prepend a module into the `User` class you would use +the following approach: ```ruby class User < ActiveRecord::Base # ... lots of code here ... end -User.prepend(EE::User) +User.prepend_if_ee('EE::User') ``` +Do not use methods such as `prepend`, `extend`, and `include`. Instead, use +`prepend_if_ee`, `extend_if_ee`, or `include_if_ee`. These methods take a +_String_ containing the full module name as the argument, not the module itself. + Since the module would require an `EE` namespace, the file should also be put in an `ee/` sub-directory. For example, we want to extend the user model in EE, so we have a module called `::EE::User` put inside @@ -255,7 +259,7 @@ class ApplicationController < ActionController::Base # ... end -ApplicationController.prepend(EE::ApplicationController) +ApplicationController.prepend_if_ee('EE::ApplicationController') ``` And create a new file in the `ee/` sub-directory with the altered @@ -504,9 +508,9 @@ EE-specific LDAP classes in `ee/lib/ee/gitlab/ldap`. ### Code in `lib/api/` -It can be very tricky to extend EE features by a single line of `prepend`, -and for each different [Grape](https://github.com/ruby-grape/grape) feature, -we might need different strategies to extend it. To apply different strategies +It can be very tricky to extend EE features by a single line of `prepend_if_ee`, +and for each different [Grape](https://github.com/ruby-grape/grape) feature, we +might need different strategies to extend it. To apply different strategies easily, we would use `extend ActiveSupport::Concern` in the EE module. Put the EE module files following @@ -543,12 +547,12 @@ constants. We can define `params` and utilize `use` in another `params` definition to include params defined in EE. However, we need to define the "interface" first in CE in order for EE to override it. We don't have to do this in other places -due to `prepend`, but Grape is complex internally and we couldn't easily do -that, so we'll follow regular object-oriented practices that we define the +due to `prepend_if_ee`, but Grape is complex internally and we couldn't easily +do that, so we'll follow regular object-oriented practices that we define the interface first here. For example, suppose we have a few more optional params for EE. We can move the -params out of the `Grape::API` class to a helper module, so we can `prepend` it +params out of the `Grape::API` class to a helper module, so we can inject it before it would be used in the class. ```ruby @@ -583,7 +587,7 @@ module API end end -API::Helpers::ProjectsHelpers.prepend(EE::API::Helpers::ProjectsHelpers) +API::Helpers::ProjectsHelpers.prepend_if_ee('EE::API::Helpers::ProjectsHelpers') ``` We could override it in EE module: @@ -624,7 +628,7 @@ module API end end -API::JobArtifacts.prepend(EE::API::JobArtifacts) +API::JobArtifacts.prepend_if_ee('EE::API::JobArtifacts') ``` And then we can follow regular object-oriented practices to override it: @@ -677,7 +681,7 @@ module API end end -API::MergeRequests.prepend(EE::API::MergeRequests) +API::MergeRequests.prepend_if_ee('EE::API::MergeRequests') ``` Note that `update_merge_request_ee` doesn't do anything in CE, but @@ -717,8 +721,8 @@ Sometimes we need to use different arguments for a particular API route, and we can't easily extend it with an EE module because Grape has different context in different blocks. In order to overcome this, we need to move the data to a class method that resides in a separate module or class. This allows us to extend that -module or class before its data is used, without having to place a `prepend` in -the middle of CE code. +module or class before its data is used, without having to place a +`prepend_if_ee` in the middle of CE code. For example, in one place we need to pass an extra argument to `at_least_one_of` so that the API could consider an EE-only argument as the @@ -739,7 +743,7 @@ module API end end -API::MergeRequests::Parameters.prepend(EE::API::MergeRequests::Parameters) +API::MergeRequests::Parameters.prepend_if_ee('EE::API::MergeRequests::Parameters') # api/merge_requests.rb module API @@ -789,7 +793,7 @@ class Identity < ActiveRecord::Base [:provider] end - prepend EE::Identity + prepend_if_ee('EE::Identity') validates :extern_uid, allow_blank: true, @@ -841,7 +845,7 @@ class Identity < ActiveRecord::Base end end -Identity::UniquenessScopes.prepend(EE::Identity::UniquenessScopes) +Identity::UniquenessScopes.prepend_if_ee('EE::Identity::UniquenessScopes') # app/models/identity.rb class Identity < ActiveRecord::Base @@ -906,7 +910,7 @@ import bundle from 'ee_else_ce/protected_branches/protected_branches_bundle.js'; ``` See the frontend guide [performance section](fe_guide/performance.md) for -information on managing page-specific javascript within EE. +information on managing page-specific JavaScript within EE. ## Vue code in `assets/javascript` @@ -941,7 +945,7 @@ export default { - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin -##### Example: +##### Example ```javascript import mixin from 'ee_else_ce/path/mixin'; @@ -972,7 +976,7 @@ For regular JS files, the approach is similar. 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: -#### Example: +#### Example ```javascript import eeCode from 'ee_else_ce/ee_code'; @@ -1053,7 +1057,7 @@ Here is a workflow to make sure those changes end up backported safely into CE t **Note:** regarding SCSS, make sure the files living outside `/ee/` don't diverge between CE and EE projects. -## gitlab-svgs +## GitLab-svgs Conflicts in `app/assets/images/icons.json` or `app/assets/images/icons.svg` can be resolved simply by regenerating those assets with diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 0965db29557..f2412c249c1 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -40,9 +40,11 @@ There is no need to install any plugins If you're interested on working with the new beta repo indexer, all you need to do is: -- git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git -- make -- make install +```sh +git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git +make +make install +``` this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that is in your `$PATH`. After that GitLab will find it and you'll be able to enable it in the admin settings area. @@ -148,6 +150,59 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/ - Searches can have their own analyzers. Remember to check when editing analyzers - `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches +## Zero downtime reindexing with multiple indices + +Currently GitLab can only handle a single version of setting. Any setting/schema changes would require reindexing everything from scratch. Since reindexing can take a long time, this can cause search functionality downtime. + +To avoid downtime, GitLab is working to support multiple indices that +can function at the same time. Whenever the schema changes, the admin +will be able to create a new index and reindex to it, while searches +continue to go to the older, stable index. Any data updates will be +forwarded to both indices. Once the new index is ready, an admin can +mark it active, which will direct all searches to it, and remove the old +index. + +This is also helpful for migrating to new servers, e.g. moving to/from AWS. + +Currently we are on the process of migrating to this new design. Everything is hardwired to work with one single version for now. + +### Architecture + +The traditional setup, provided by `elasticsearch-rails`, is to communicate through its internal proxy classes. Developers would write model-specific logic in a module for the model to include in (e.g. `SnippetsSearch`). The `__elasticsearch__` methods would return a proxy object, e.g.: + +- `Issue.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy` +- `Issue.first.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. + +These proxy objects would talk to Elasticsearch server directly (see top half of the diagram). + + + +In the planned new design, each model would have a pair of corresponding subclassed proxy objects, in which model-specific logic is located. For example, `Snippet` would have `SnippetClassProxy` and `SnippetInstanceProxy` (being subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy` and `Elasticsearch::Model::Proxy::InstanceMethodsProxy`, respectively). + +`__elasticsearch__` would represent another layer of proxy object, keeping track of multiple actual proxy objects. It would forward method calls to the appropriate index. For example: + +- `model.__elasticsearch__.search` would be forwarded to the one stable index, since it is a read operation. +- `model.__elasticsearch__.update_document` would be forwarded to all indices, to keep all indices up-to-date. + +The global configurations per version are now in the `Elastic::(Version)::Config` class. You can change mappings there. + +### Creating new version of schema + +NOTE: **Note:** this is not applicable yet as multiple indices functionality is not fully implemented. + +Folders like `ee/lib/elastic/v12p1` contain snapshots of search logic from different versions. To keep a continuous Git history, the latest version lives under `ee/lib/elastic/latest`, but its classes are aliased under an actual version (e.g. `ee/lib/elastic/v12p3`). When referencing these classes, never use the `Latest` namespace directly, but use the actual version (e.g. `V12p3`). + +The version name basically follows GitLab's release version. If setting is changed in 12.3, we will create a new namespace called `V12p3` (p stands for "point"). Raise an issue if there is a need to name a version differently. + +If the current version is `v12p1`, and we need to create a new version for `v12p3`, the steps are as follows: + +1. Copy the entire folder of `v12p1` as `v12p3` +1. Change the namespace for files under `v12p3` folder from `V12p1` to `V12p3` (which are still aliased to `Latest`) +1. Delete `v12p1` folder +1. Copy the entire folder of `latest` as `v12p1` +1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1` +1. Make changes to files under the `latest` folder as needed + ## Troubleshooting ### Getting `flood stage disk watermark [95%] exceeded` diff --git a/doc/development/emails.md b/doc/development/emails.md index e6af075a282..5676c3b32f4 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -5,6 +5,10 @@ To view rendered emails "sent" in your development instance, visit [`/rails/letter_opener`](http://localhost:3000/rails/letter_opener). +Please note that [S/MIME signed](../administration/smime_signing_email.md) emails +[cannot be currently previewed](https://github.com/fgrehm/letter_opener_web/issues/96) with +`letter_opener`. + ## Mailer previews Rails provides a way to preview our mailer templates in HTML and plaintext using diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 49b74b5ebcf..3d27f67a8a6 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -11,7 +11,7 @@ Architectural decisions should be accessible to everyone, so please document them in the relevant Merge Request discussion or by updating our documentation when appropriate. -You can find the Frontend Architecture experts on the [team page](https://about.gitlab.com/company/team). +You can find the Frontend Architecture experts on the [team page](https://about.gitlab.com/company/team/). ## Examples diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index 0d9397c3bd5..6e7cf523f36 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -1,15 +1,18 @@ # Axios + We use [axios][axios] to communicate with the server in Vue applications and most new code. In order to guarantee all defaults are set you *should not use `axios` directly*, you should import `axios` from `axios_utils`. ## CSRF token + All our request require a CSRF token. To guarantee this token is set, we are importing [axios][axios], setting the token, and exporting `axios` . This exported module should be used instead of directly using `axios` to ensure the token is set. ## Usage + ```javascript import axios from './lib/utils/axios_utils'; @@ -35,7 +38,7 @@ Advantages over [`spyOn()`]: - no need to create response objects - does not allow call through (which we want to avoid) -- simple API to test error cases +- simple API to test error cases - provides `replyOnce()` to allow for different responses We have also decided against using [axios interceptors] because they are not suitable for mocking. diff --git a/doc/development/fe_guide/components.md b/doc/development/fe_guide/components.md index 52462a4bec9..096ce8ca25a 100644 --- a/doc/development/fe_guide/components.md +++ b/doc/development/fe_guide/components.md @@ -14,28 +14,29 @@ See also the [corresponding UX guide](https://design.gitlab.com/#/components/dro 1. Use the HTML structure provided by the [docs][bootstrap-dropdowns] 1. Add a specific class to the top level `.dropdown` element - ```Haml - .dropdown.my-dropdown - %button{ type: 'button', data: { toggle: 'dropdown' }, 'aria-haspopup': true, 'aria-expanded': false } - %span.dropdown-toggle-text - Toggle Dropdown - = icon('chevron-down') - - %ul.dropdown-menu - %li - %a - item! - ``` - - Or use the helpers - ```Haml - .dropdown.my-dropdown - = dropdown_toggle('Toogle!', { toggle: 'dropdown' }) - = dropdown_content - %li - %a - item! - ``` + ```Haml + .dropdown.my-dropdown + %button{ type: 'button', data: { toggle: 'dropdown' }, 'aria-haspopup': true, 'aria-expanded': false } + %span.dropdown-toggle-text + Toggle Dropdown + = icon('chevron-down') + + %ul.dropdown-menu + %li + %a + item! + ``` + + Or use the helpers + + ```Haml + .dropdown.my-dropdown + = dropdown_toggle('Toogle!', { toggle: 'dropdown' }) + = dropdown_content + %li + %a + item! + ``` [bootstrap-dropdowns]: https://getbootstrap.com/docs/3.3/javascript/#dropdowns diff --git a/doc/development/fe_guide/design_patterns.md b/doc/development/fe_guide/design_patterns.md index 0342d16a87c..2f372f783f5 100644 --- a/doc/development/fe_guide/design_patterns.md +++ b/doc/development/fe_guide/design_patterns.md @@ -53,6 +53,7 @@ When writing a class that needs to manipulate the DOM guarantee a container opti This is useful when we need that class to be instantiated more than once in the same page. Bad: + ```javascript class Foo { constructor() { @@ -63,6 +64,7 @@ new Foo(); ``` Good: + ```javascript class Foo { constructor(opts) { @@ -72,6 +74,7 @@ class Foo { new Foo({ container: '.my-element' }); ``` + You can find an example of the above in this [class][container-class-example]; [container-class-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/mini_pipeline_graph_dropdown.js diff --git a/doc/development/fe_guide/droplab/droplab.md b/doc/development/fe_guide/droplab/droplab.md index 2f8c79abde1..1c6d895b3ab 100644 --- a/doc/development/fe_guide/droplab/droplab.md +++ b/doc/development/fe_guide/droplab/droplab.md @@ -25,6 +25,7 @@ If you do not provide any arguments, it will globally query and instantiate all <!-- ... --> <ul> ``` + ```js const droplab = new DropLab(); droplab.init(); @@ -45,6 +46,7 @@ You can add static list items. <li>Static value 2</li> <ul> ``` + ```js const droplab = new DropLab(); droplab.init(); @@ -62,6 +64,7 @@ a non-global instance of DropLab using the `DropLab.prototype.init` method. <!-- ... --> <ul> ``` + ```js const trigger = document.getElementById('trigger'); const list = document.getElementById('list'); @@ -79,6 +82,7 @@ You can also add hooks to an existing DropLab instance using `DropLab.prototype. <a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> <ul id="list" data-dropdown><!-- ... --><ul> ``` + ```js const droplab = new DropLab(); @@ -109,6 +113,7 @@ for all `data-dynamic` dropdown lists tracked by that DropLab instance. <li><a href="#" data-id="{{id}}">{{text}}</a></li> </ul> ``` + ```js const droplab = new DropLab(); @@ -131,6 +136,7 @@ the data as the second argument and the `id` of the trigger element as the first <li><a href="#" data-id="{{id}}">{{text}}</a></li> </ul> ``` + ```js const droplab = new DropLab(); @@ -160,6 +166,7 @@ dropdown lists, one of which is dynamic. </ul> </div> ``` + ```js const droplab = new DropLab(); @@ -216,6 +223,7 @@ Some plugins require configuration values, the config object can be passed as th <a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> <ul id="list" data-dropdown><!-- ... --><ul> ``` + ```js const droplab = new DropLab(); diff --git a/doc/development/fe_guide/droplab/plugins/ajax.md b/doc/development/fe_guide/droplab/plugins/ajax.md index b6a883ce6c4..4b76b207d88 100644 --- a/doc/development/fe_guide/droplab/plugins/ajax.md +++ b/doc/development/fe_guide/droplab/plugins/ajax.md @@ -17,18 +17,19 @@ Add the `Ajax` object to the plugins array of a `DropLab.prototype.init` or `Dro <a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> <ul id="list" data-dropdown><!-- ... --><ul> ``` + ```js - const droplab = new DropLab(); +const droplab = new DropLab(); - const trigger = document.getElementById('trigger'); - const list = document.getElementById('list'); +const trigger = document.getElementById('trigger'); +const list = document.getElementById('list'); - droplab.addHook(trigger, list, [Ajax], { - Ajax: { - endpoint: '/some-endpoint', - method: 'setData', - }, - }); +droplab.addHook(trigger, list, [Ajax], { + Ajax: { + endpoint: '/some-endpoint', + method: 'setData', + }, +}); ``` Optionally you can set `loadingTemplate` to a HTML string. This HTML string will diff --git a/doc/development/fe_guide/droplab/plugins/filter.md b/doc/development/fe_guide/droplab/plugins/filter.md index 1f188c64fe4..b867394a241 100644 --- a/doc/development/fe_guide/droplab/plugins/filter.md +++ b/doc/development/fe_guide/droplab/plugins/filter.md @@ -17,25 +17,26 @@ Add the `Filter` object to the plugins array of a `DropLab.prototype.init` or `D <li><a href="#" data-id="{{id}}">{{text}}</a></li> <ul> ``` + ```js - const droplab = new DropLab(); - - const trigger = document.getElementById('trigger'); - const list = document.getElementById('list'); - - droplab.init(trigger, list, [Filter], { - Filter: { - template: 'text', - }, - }); - - droplab.addData('trigger', [{ - id: 0, - text: 'Jacob', - }, { - id: 1, - text: 'Jeff', - }]); +const droplab = new DropLab(); + +const trigger = document.getElementById('trigger'); +const list = document.getElementById('list'); + +droplab.init(trigger, list, [Filter], { + Filter: { + template: 'text', + }, +}); + +droplab.addData('trigger', [{ + id: 0, + text: 'Jacob', +}, { + id: 1, + text: 'Jeff', +}]); ``` Above, the input string will be compared against the `test` key of the passed data objects. diff --git a/doc/development/fe_guide/droplab/plugins/input_setter.md b/doc/development/fe_guide/droplab/plugins/input_setter.md index e4050213869..db492da478a 100644 --- a/doc/development/fe_guide/droplab/plugins/input_setter.md +++ b/doc/development/fe_guide/droplab/plugins/input_setter.md @@ -22,33 +22,34 @@ You can also set the `InputSetter` config to an array of objects, which will all <li><a href="#" data-id="{{id}}">{{text}}</a></li> <ul> ``` + ```js - const droplab = new DropLab(); - - const trigger = document.getElementById('trigger'); - const list = document.getElementById('list'); - - const input = document.getElementById('input'); - const div = document.getElementById('div'); - - droplab.init(trigger, list, [InputSetter], { - InputSetter: [{ - input: input, - valueAttribute: 'data-id', - } { - input: div, - valueAttribute: 'data-id', - inputAttribute: 'data-selected-id', - }], - }); - - droplab.addData('trigger', [{ - id: 0, - text: 'Jacob', - }, { - id: 1, - text: 'Jeff', - }]); +const droplab = new DropLab(); + +const trigger = document.getElementById('trigger'); +const list = document.getElementById('list'); + +const input = document.getElementById('input'); +const div = document.getElementById('div'); + +droplab.init(trigger, list, [InputSetter], { + InputSetter: [{ + input: input, + valueAttribute: 'data-id', + } { + input: div, + valueAttribute: 'data-id', + inputAttribute: 'data-selected-id', + }], +}); + +droplab.addData('trigger', [{ + id: 0, + text: 'Jacob', +}, { + id: 1, + text: 'Jeff', +}]); ``` Above, if the second list item was clicked, it would update the `#input` element diff --git a/doc/development/fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md index 6ab3fa4acf3..1b417d4c8c2 100644 --- a/doc/development/fe_guide/event_tracking.md +++ b/doc/development/fe_guide/event_tracking.md @@ -1,79 +1,76 @@ -# Event Tracking +# Event tracking -We use [Snowplow](https://github.com/snowplow/snowplow) for tracking custom events (available in GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) only). +GitLab provides `Tracking`, an interface that wraps +[Snowplow](https://github.com/snowplow/snowplow) for tracking custom events. +It uses Snowplow's custom event tracking functions. -## Generic tracking function - -In addition to Snowplow's built-in method for tracking page views, we use a generic tracking function which enables us to selectively apply listeners to events. - -The generic tracking function can be imported in EE-specific JS files as follows: +The tracking interface can be imported in JS files as follows: ```javascript -import { trackEvent } from `ee/stats`; +import Tracking from `~/tracking`; ``` -This gives the user access to the `trackEvent` method, which takes the following parameters: +## Tracking in HAML or Vue templates -| parameter | type | description | required | -| ---------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -| `category` | string | Describes the page that you're capturing click events on. Unless infeasible, please use the Rails page attribute `document.body.dataset.page` by default. | true | -| `eventName` | string | Describes the action the user is taking. The first word should always describe the action. For example, clicks should be `click` and activations should be `activate`. Use underscores to describe what was acted on. For example, activating a form field would be `activate_form_input`. Clicking on a dropdown is `click_dropdown`. | true | -| `additionalData` | object | Additional data such as `label`, `property`, and `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | false | +To avoid having to do create a bunch of custom javascript event handlers, when working within HAML or Vue templates, we can add `data-track-*` attributes to elements of interest. This way, all elements that have a `data-track-event` attribute to automatically have event tracking bound. -Read more about instrumentation and the taxonomy in the [Product Handbook](https://about.gitlab.com/handbook/product/feature-instrumentation). - -### Tracking in `.js` and `.vue` files +Below is an example of `data-track-*` attributes assigned to a button in HAML: -The most simple use case is to add tracking programmatically to an event of interest in Javascript. +```haml +%button.btn{ data: { track_event: "click_button", track_label: "template_preview", track_property: "my-template", track_value: "" } } +``` -The following example demonstrates how to track a click on a button in Javascript by calling the `trackEvent` method explicitly: +We can then setup tracking for large sections of a page, or an entire page by telling the Tracking interface to bind to it. ```javascript -import { trackEvent } from `ee/stats`; +import Tracking from '~/tracking'; -trackEvent('dashboard:projects:index', 'click_button', { - label: 'create_from_template', - property: 'template_preview', - value: 'rails', +// for the entire document +new Tracking().bind(); + +// for a container element +document.addEventListener('DOMContentLoaded', () => { + new Tracking('my_category').bind(document.getElementById('my-container')); }); + ``` -### Tracking in HAML templates +When you instantiate a Tracking instance you can provide a category. If none is provided, `document.body.dataset.page` will be used. When you bind the Tracking instance you can provide an element. If no element is provided to bind to, the `document` is assumed. -Sometimes we want to track clicks for multiple elements on a page. Creating event handlers for all elements could soon turn into a tedious task. +Below is a list of supported `data-track-*` attributes: -There's a more convenient solution to this problem. When working with HAML templates, we can add `data-track-*` attributes to elements of interest. This way, all elements that have both `data-track-label` and `data-track-event` attributes assigned get marked for event tracking. All we have to do is call the `bindTrackableContainer` method on a container which allows for better scoping. +| attribute | required | description | +|:----------------------|:---------|:------------| +| `data-track-event` | true | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | +| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | +| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | +| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). If omitted, this will be the elements `value` property or an empty string. For checkboxes, the default value will be the element's checked attribute or `false` when unchecked. | -Below is an example of `data-track-*` attributes assigned to a button in HAML: +## Tracking in raw Javascript -```ruby -%button.btn{ data: { track_label: "create_from_template", track_property: "template_preview", track_event: "click_button", track_value: "my-template" } } -``` +Custom events can be tracked by directly calling the `Tracking.event` static function, which accepts the following arguments: + +| argument | type | default value | description | +|:-----------|:-------|:---------------------------|:------------| +| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. | +| `event` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | +| `data` | object | {} | Additional data such as `label`, `property`, and `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). These will be set as empty strings if you don't provide them. | -By calling `bindTrackableContainer('.my-container')`, click handlers get bound to all elements located in `.my-container` provided that they have the necessary `data-track-*` attributes assigned to them. +Tracking can be programmatically added to an event of interest in Javascript, and the following example demonstrates tracking a click on a button by calling `Tracking.event` manually. ```javascript -import Stats from 'ee/stats'; +import Tracking from `~/tracking`; -document.addEventListener('DOMContentLoaded', () => { - Stats.bindTrackableContainer('.my-container', 'category'); -}); +document.getElementById('my_button').addEventListener('click', () => { + Tracking.event('dashboard:projects:index', 'click_button', { + label: 'create_from_template', + property: 'template_preview', + value: 'rails', + }); +}) ``` -The second parameter in `bindTrackableContainer` is optional. If omitted, the value of `document.body.dataset.page` will be used as category instead. - -Below is a list of supported `data-track-*` attributes: - -| attribute | description | required | -| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | -| `data-track-label` | The `label` in `trackEvent` | true | -| `data-track-event` | The `eventName` in `trackEvent` | true | -| `data-track-property` | The `property` in `trackEvent`. If omitted, an empty string will be used as a default value. | false | -| `data-track-value` | The `value` in `trackEvent`. If omitted, this will be `target.value` or empty string. For checkboxes, the default value being tracked will be the element's checked attribute if `data-track-value` is omitted. | false | - -Since Snowplow is an Enterprise Edition feature, it's necessary to create a CE backport when adding `data-track-*` attributes to HAML templates in most cases. - -## Testing +## Toggling tracking on or off Snowplow can be enabled by navigating to: diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index e4225f2bc39..0d2aeffeac0 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -5,12 +5,12 @@ 1. **You talk about Frontend FAQ.** Please share links to it whenever applicable, so more eyes catch when content gets outdated. -2. **Keep it short and simple.** +1. **Keep it short and simple.** Whenever an answer needs more than two sentences it does not belong here. -3. **Provide background when possible.** +1. **Provide background when possible.** Linking to relevant source code, issue / epic, or other documentation helps to understand the answer. -4. **If you see something, do something.** +1. **If you see something, do something.** Please remove or update any content that is outdated as soon as you see it. ## FAQ diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 55b719227e5..4fc5dfc8c3d 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -47,7 +47,7 @@ new Vue({ }); ``` -Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation][vue-apollo-docs]. +Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation](https://vue-apollo.netlify.com/guide/). ### Local state with Apollo @@ -118,7 +118,6 @@ Read more about the [Apollo] client in the [Apollo documentation](https://www.ap [Apollo]: https://www.apollographql.com/ [vue-apollo]: https://github.com/Akryum/vue-apollo/ -[vue-apollo-docs]: https://akryum.github.io/vue-apollo/ [feature-flags]: ../feature_flags.md [default-client]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/lib/graphql.js [vue-test-utils]: https://vue-test-utils.vuejs.org/ diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index 533e2001300..4f687d8642e 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -21,10 +21,10 @@ To use a sprite Icon in HAML or Rails we use a specific helper function : sprite_icon(icon_name, size: nil, css_class: '') ``` -- **icon_name** Use the icon_name that you can find in the SVG Sprite - ([Overview is available here][svg-preview]). -- **size (optional)** Use one of the following sizes : 16, 24, 32, 48, 72 (this will be translated into a `s16` class) -- **css_class (optional)** If you want to add additional css classes +- **icon_name** Use the icon_name that you can find in the SVG Sprite + ([Overview is available here][svg-preview]). +- **size (optional)** Use one of the following sizes : 16, 24, 32, 48, 72 (this will be translated into a `s16` class) +- **css_class (optional)** If you want to add additional css classes **Example** @@ -65,10 +65,10 @@ export default { </template> ``` -- **name** Name of the Icon in the SVG Sprite ([Overview is available here][svg-preview]). -- **size (optional)** Number value for the size which is then mapped to a specific CSS class - (Available Sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped to `sXX` css classes) -- **css-classes (optional)** Additional CSS Classes to add to the svg tag. +- **name** Name of the Icon in the SVG Sprite ([Overview is available here][svg-preview]). +- **size (optional)** Number value for the size which is then mapped to a specific CSS class + (Available Sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped to `sXX` css classes) +- **css-classes (optional)** Additional CSS Classes to add to the svg tag. ### Usage in HTML/JS diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 36d5e4ab96b..deaef8e768b 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -17,7 +17,7 @@ Working with our frontend assets requires Node (v8.10.0 or greater) and Yarn For our currently-supported browsers, see our [requirements][requirements]. ---- +Use [BrowserStack](https://www.browserstack.com/) to test with our supported browsers. Login to BrowserStack with the credentials saved in GitLab's [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). ## Initiatives @@ -77,8 +77,6 @@ How we use Snowplow to track custom events. Read the [frontend's FAQ](frontend_faq.md) for common small pieces of helpful information. ---- - ## Style Guides ### [JavaScript Style Guide](style_guide_js.md) @@ -91,20 +89,14 @@ changes. Our SCSS conventions which are enforced through [scss-lint][scss-lint]. ---- - ## [Performance](performance.md) Best practices for monitoring and maximizing frontend performance. ---- - ## [Security](security.md) Frontend security practices. ---- - ## [Accessibility](accessibility.md) Our accessibility standards and resources. diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 2628e95dbc1..676bce32998 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -30,8 +30,8 @@ To improve the time to first render we are using lazy loading for images. This w the actual image source on the `data-src` attribute. After the HTML is rendered and JavaScript is loaded, the value of `data-src` will be moved to `src` automatically if the image is in the current viewport. -- Prepare images in HTML for lazy loading by renaming the `src` attribute to `data-src` AND adding the class `lazy`. -- If you are using the Rails `image_tag` helper, all images will be lazy-loaded by default unless `lazy: false` is provided. +- Prepare images in HTML for lazy loading by renaming the `src` attribute to `data-src` AND adding the class `lazy`. +- If you are using the Rails `image_tag` helper, all images will be lazy-loaded by default unless `lazy: false` is provided. If you are asynchronously adding content which contains lazy images then you need to call the function `gl.lazyLoader.searchLazyImages()` which will search for lazy images and load them if needed. @@ -96,26 +96,26 @@ bundle and included on the page. DOM has loaded, you should attach an event handler to the `DOMContentLoaded` event with: - ```javascript - import initMyWidget from './my_widget'; + ```javascript + import initMyWidget from './my_widget'; - document.addEventListener('DOMContentLoaded', () => { - initMyWidget(); - }); - ``` + document.addEventListener('DOMContentLoaded', () => { + initMyWidget(); + }); + ``` - **Supporting Module Placement:** - - If a class or a module is _specific to a particular route_, try to locate - it close to the entry point it will be used. For instance, if - `my_widget.js` is only imported within `pages/widget/show/index.js`, you - should place the module at `pages/widget/show/my_widget.js` and import it - with a relative path (e.g. `import initMyWidget from './my_widget';`). - - If a class or module is _used by multiple routes_, place it within a - shared directory at the closest common parent directory for the entry - points that import it. For example, if `my_widget.js` is imported within - both `pages/widget/show/index.js` and `pages/widget/run/index.js`, then - place the module at `pages/widget/shared/my_widget.js` and import it with - a relative path if possible (e.g. `../shared/my_widget`). + - If a class or a module is _specific to a particular route_, try to locate + it close to the entry point it will be used. For instance, if + `my_widget.js` is only imported within `pages/widget/show/index.js`, you + should place the module at `pages/widget/show/my_widget.js` and import it + with a relative path (e.g. `import initMyWidget from './my_widget';`). + - If a class or module is _used by multiple routes_, place it within a + shared directory at the closest common parent directory for the entry + points that import it. For example, if `my_widget.js` is imported within + both `pages/widget/show/index.js` and `pages/widget/run/index.js`, then + place the module at `pages/widget/shared/my_widget.js` and import it with + a relative path if possible (e.g. `../shared/my_widget`). - **Enterprise Edition Caveats:** For GitLab Enterprise Edition, page-specific entry points will override their @@ -161,7 +161,7 @@ General tips: - Use code-splitting dynamic imports wherever possible to lazy-load code that is not needed initially. - [High Performance Animations][high-perf-animations] -------- +--- ## Additional Resources diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index b50159c2b75..125b11afcd0 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -20,148 +20,148 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. **Never Ever EVER** disable eslint globally for a file - ```javascript - // bad - /* eslint-disable */ + ```javascript + // bad + /* eslint-disable */ - // better - /* eslint-disable some-rule, some-other-rule */ + // better + /* eslint-disable some-rule, some-other-rule */ - // best - // nothing :) - ``` + // best + // nothing :) + ``` 1. If you do need to disable a rule for a single violation, try to do it as locally as possible - ```javascript - // bad - /* eslint-disable no-new */ + ```javascript + // bad + /* eslint-disable no-new */ - import Foo from 'foo'; + import Foo from 'foo'; - new Foo(); + new Foo(); - // better - import Foo from 'foo'; + // better + import Foo from 'foo'; - // eslint-disable-next-line no-new - new Foo(); - ``` + // eslint-disable-next-line no-new + new Foo(); + ``` 1. There are few rules that we need to disable due to technical debt. Which are: - 1. [no-new][eslint-new] - 1. [class-methods-use-this][eslint-this] + 1. [no-new](https://eslint.org/docs/rules/no-new) + 1. [class-methods-use-this](https://eslint.org/docs/rules/class-methods-use-this) 1. When they are needed _always_ place ESlint directive comment blocks on the first line of a script, followed by any global declarations, then a blank newline prior to any imports or code. - ```javascript - // bad - /* global Foo */ - /* eslint-disable no-new */ - import Bar from './bar'; + ```javascript + // bad + /* global Foo */ + /* eslint-disable no-new */ + import Bar from './bar'; - // good - /* eslint-disable no-new */ - /* global Foo */ + // good + /* eslint-disable no-new */ + /* global Foo */ - import Bar from './bar'; - ``` + import Bar from './bar'; + ``` 1. **Never** disable the `no-undef` rule. Declare globals with `/* global Foo */` instead. 1. When declaring multiple globals, always use one `/* global [name] */` line per variable. - ```javascript - // bad - /* globals Flash, Cookies, jQuery */ + ```javascript + // bad + /* globals Flash, Cookies, jQuery */ - // good - /* global Flash */ - /* global Cookies */ - /* global jQuery */ - ``` + // good + /* global Flash */ + /* global Cookies */ + /* global jQuery */ + ``` 1. Use up to 3 parameters for a function or class. If you need more accept an Object instead. - ```javascript - // bad - fn(p1, p2, p3, p4) {} + ```javascript + // bad + fn(p1, p2, p3, p4) {} - // good - fn(options) {} - ``` + // good + fn(options) {} + ``` #### Modules, Imports, and Exports 1. Use ES module syntax to import modules - ```javascript - // bad - const SomeClass = require('some_class'); + ```javascript + // bad + const SomeClass = require('some_class'); - // good - import SomeClass from 'some_class'; + // good + import SomeClass from 'some_class'; - // bad - module.exports = SomeClass; + // bad + module.exports = SomeClass; - // good - export default SomeClass; - ``` + // good + export default SomeClass; + ``` - Import statements are following usual naming guidelines, for example object literals use camel case: + Import statements are following usual naming guidelines, for example object literals use camel case: - ```javascript - // some_object file - export default { - key: 'value', - }; + ```javascript + // some_object file + export default { + key: 'value', + }; - // bad - import ObjectLiteral from 'some_object'; + // bad + import ObjectLiteral from 'some_object'; - // good - import objectLiteral from 'some_object'; - ``` + // good + import objectLiteral from 'some_object'; + ``` 1. Relative paths: when importing a module in the same directory, a child directory, or an immediate parent directory prefer relative paths. When importing a module which is two or more levels up, prefer either `~/` or `ee/`. - In **app/assets/javascripts/my-feature/subdir**: + In **app/assets/javascripts/my-feature/subdir**: - ```javascript - // bad - import Foo from '~/my-feature/foo'; - import Bar from '~/my-feature/subdir/bar'; - import Bin from '~/my-feature/subdir/lib/bin'; + ```javascript + // bad + import Foo from '~/my-feature/foo'; + import Bar from '~/my-feature/subdir/bar'; + import Bin from '~/my-feature/subdir/lib/bin'; - // good - import Foo from '../foo'; - import Bar from './bar'; - import Bin from './lib/bin'; - ``` + // good + import Foo from '../foo'; + import Bar from './bar'; + import Bin from './lib/bin'; + ``` - In **spec/javascripts**: + In **spec/javascripts**: - ```javascript - // bad - import Foo from '../../app/assets/javascripts/my-feature/foo'; + ```javascript + // bad + import Foo from '../../app/assets/javascripts/my-feature/foo'; - // good - import Foo from '~/my-feature/foo'; - ``` + // good + import Foo from '~/my-feature/foo'; + ``` - When referencing an **EE component**: + When referencing an **EE component**: - ```javascript - // bad - import Foo from '../../../../../ee/app/assets/javascripts/my-feature/ee-foo'; + ```javascript + // bad + import Foo from '../../../../../ee/app/assets/javascripts/my-feature/ee-foo'; - // good - import Foo from 'ee/my-feature/foo'; - ``` + // good + import Foo from 'ee/my-feature/foo'; + ``` 1. Avoid using IIFE. Although we have a lot of examples of files which wrap their contents in IIFEs (immediately-invoked function expressions), @@ -170,136 +170,136 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ 1. Avoid adding to the global namespace. - ```javascript - // bad - window.MyClass = class { /* ... */ }; + ```javascript + // bad + window.MyClass = class { /* ... */ }; - // good - export default class MyClass { /* ... */ } - ``` + // good + export default class MyClass { /* ... */ } + ``` 1. Side effects are forbidden in any script which contains export - ```javascript - // bad - export default class MyClass { /* ... */ } + ```javascript + // bad + export default class MyClass { /* ... */ } - document.addEventListener("DOMContentLoaded", function(event) { - new MyClass(); - } - ``` + document.addEventListener("DOMContentLoaded", function(event) { + new MyClass(); + } + ``` #### Data Mutation and Pure functions 1. Strive to write many small pure functions, and minimize where mutations occur. - ```javascript - // bad - const values = {foo: 1}; + ```javascript + // bad + const values = {foo: 1}; - function impureFunction(items) { - const bar = 1; + function impureFunction(items) { + const bar = 1; - items.foo = items.a * bar + 2; + items.foo = items.a * bar + 2; - return items.a; - } + return items.a; + } - const c = impureFunction(values); + const c = impureFunction(values); - // good - var values = {foo: 1}; + // good + var values = {foo: 1}; - function pureFunction (foo) { - var bar = 1; + function pureFunction (foo) { + var bar = 1; - foo = foo * bar + 2; + foo = foo * bar + 2; - return foo; - } + return foo; + } - var c = pureFunction(values.foo); + var c = pureFunction(values.foo); ``` 1. Avoid constructors with side-effects. Although we aim for code without side-effects we need some side-effects for our code to run. - If the class won't do anything if we only instantiate it, it's ok to add side effects into the constructor (_Note:_ The following is just an example. If the only purpose of the class is to add an event listener and handle the callback a function will be more suitable.) - - ```javascript - // Bad - export class Foo { - constructor() { - this.init(); - } - init() { - document.addEventListener('click', this.handleCallback) - }, - handleCallback() { - - } - } - - // Good - export class Foo { - constructor() { - document.addEventListener() - } - handleCallback() { - } - } - ``` - - On the other hand, if a class only needs to extend a third party/add event listeners in some specific cases, they should be initialized outside of the constructor. + If the class won't do anything if we only instantiate it, it's ok to add side effects into the constructor (_Note:_ The following is just an example. If the only purpose of the class is to add an event listener and handle the callback a function will be more suitable.) + + ```javascript + // Bad + export class Foo { + constructor() { + this.init(); + } + init() { + document.addEventListener('click', this.handleCallback) + }, + handleCallback() { + + } + } + + // Good + export class Foo { + constructor() { + document.addEventListener() + } + handleCallback() { + } + } + ``` + + On the other hand, if a class only needs to extend a third party/add event listeners in some specific cases, they should be initialized outside of the constructor. 1. Prefer `.map`, `.reduce` or `.filter` over `.forEach` A forEach will most likely cause side effects, it will be mutating the array being iterated. Prefer using `.map`, `.reduce` or `.filter` - ```javascript - const users = [ { name: 'Foo' }, { name: 'Bar' } ]; + ```javascript + const users = [ { name: 'Foo' }, { name: 'Bar' } ]; - // bad - users.forEach((user, index) => { - user.id = index; - }); + // bad + users.forEach((user, index) => { + user.id = index; + }); - // good - const usersWithId = users.map((user, index) => { - return Object.assign({}, user, { id: index }); - }); - ``` + // good + const usersWithId = users.map((user, index) => { + return Object.assign({}, user, { id: index }); + }); + ``` #### Parse Strings into Numbers 1. `parseInt()` is preferable over `Number()` or `+` - ```javascript - // bad - +'10' // 10 + ```javascript + // bad + +'10' // 10 - // good - Number('10') // 10 + // good + Number('10') // 10 - // better - parseInt('10', 10); - ``` + // better + parseInt('10', 10); + ``` #### CSS classes used for JavaScript 1. If the class is being used in Javascript it needs to be prepend with `js-` - ```html - // bad - <button class="add-user"> - Add User - </button> + ```html + // bad + <button class="add-user"> + Add User + </button> - // good - <button class="js-add-user"> - Add User - </button> - ``` + // good + <button class="js-add-user"> + Add User + </button> + ``` ### Vue.js @@ -314,43 +314,44 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. The store has it's own file 1. Use a function in the bundle file to instantiate the Vue component: - ```javascript - // bad - class { - init() { - new Component({}) - } - } - - // good - document.addEventListener('DOMContentLoaded', () => new Vue({ - el: '#element', - components: { - componentName - }, - render: createElement => createElement('component-name'), - })); - ``` + ```javascript + // bad + class { + init() { + new Component({}) + } + } + + // good + document.addEventListener('DOMContentLoaded', () => new Vue({ + el: '#element', + components: { + componentName + }, + render: createElement => createElement('component-name'), + })); + ``` 1. Do not use a singleton for the service or the store - ```javascript - // bad - class Store { - constructor() { - if (!this.prototype.singleton) { - // do something - } - } - } + ```javascript + // bad + class Store { + constructor() { + if (!this.prototype.singleton) { + // do something + } + } + } + + // good + class Store { + constructor() { + // do something + } + } + ``` - // good - class Store { - constructor() { - // do something - } - } - ``` 1. Use `.vue` for Vue templates. Do not use `%template` in HAML. #### Naming @@ -358,38 +359,38 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension ([#34371]). 1. **Reference Naming**: Use PascalCase for their instances: - ```javascript - // bad - import cardBoard from 'cardBoard.vue' + ```javascript + // bad + import cardBoard from 'cardBoard.vue' - components: { - cardBoard, - }; + components: { + cardBoard, + }; - // good - import CardBoard from 'cardBoard.vue' + // good + import CardBoard from 'cardBoard.vue' - components: { - CardBoard, - }; - ``` + components: { + CardBoard, + }; + ``` 1. **Props Naming:** Avoid using DOM component prop names. 1. **Props Naming:** Use kebab-case instead of camelCase to provide props in templates. - ```javascript - // bad - <component class="btn"> + ```javascript + // bad + <component class="btn"> - // good - <component css-class="btn"> + // good + <component css-class="btn"> - // bad - <component myProp="prop" /> + // bad + <component myProp="prop" /> - // good - <component my-prop="prop" /> - ``` + // good + <component my-prop="prop" /> + ``` [#34371]: https://gitlab.com/gitlab-org/gitlab-ce/issues/34371 @@ -399,205 +400,205 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. 1. With more than one attribute, all attributes should be on a new line: - ```javascript - // bad - <component v-if="bar" - param="baz" /> + ```javascript + // bad + <component v-if="bar" + param="baz" /> - <button class="btn">Click me</button> + <button class="btn">Click me</button> - // good - <component - v-if="bar" - param="baz" - /> + // good + <component + v-if="bar" + param="baz" + /> - <button class="btn"> - Click me - </button> - ``` + <button class="btn"> + Click me + </button> + ``` 1. The tag can be inline if there is only one attribute: - ```javascript - // good - <component bar="bar" /> + ```javascript + // good + <component bar="bar" /> - // good - <component - bar="bar" - /> + // good + <component + bar="bar" + /> - // bad - <component - bar="bar" /> - ``` + // bad + <component + bar="bar" /> + ``` #### Quotes 1. Always use double quotes `"` inside templates and single quotes `'` for all other JS. - ```javascript - // bad - template: ` - <button :class='style'>Button</button> - ` + ```javascript + // bad + template: ` + <button :class='style'>Button</button> + ` - // good - template: ` - <button :class="style">Button</button> - ` - ``` + // good + template: ` + <button :class="style">Button</button> + ` + ``` #### Props 1. Props should be declared as an object - ```javascript - // bad - props: ['foo'] + ```javascript + // bad + props: ['foo'] - // good - props: { - foo: { - type: String, - required: false, - default: 'bar' - } - } - ``` + // good + props: { + foo: { + type: String, + required: false, + default: 'bar' + } + } + ``` 1. Required key should always be provided when declaring a prop - ```javascript - // bad - props: { - foo: { - type: String, - } - } - - // good - props: { - foo: { - type: String, - required: false, - default: 'bar' - } - } - ``` + ```javascript + // bad + props: { + foo: { + type: String, + } + } + + // good + props: { + foo: { + type: String, + required: false, + default: 'bar' + } + } + ``` 1. Default key should be provided if the prop is not required. _Note:_ There are some scenarios where we need to check for the existence of the property. On those a default key should not be provided. - ```javascript - // good - props: { - foo: { - type: String, - required: false, - } - } - - // good - props: { - foo: { - type: String, - required: false, - default: 'bar' - } - } - - // good - props: { - foo: { - type: String, - required: true - } - } - ``` + ```javascript + // good + props: { + foo: { + type: String, + required: false, + } + } + + // good + props: { + foo: { + type: String, + required: false, + default: 'bar' + } + } + + // good + props: { + foo: { + type: String, + required: true + } + } + ``` #### Data 1. `data` method should always be a function - ```javascript - // bad - data: { - foo: 'foo' - } + ```javascript + // bad + data: { + foo: 'foo' + } - // good - data() { - return { - foo: 'foo' - }; - } - ``` + // good + data() { + return { + foo: 'foo' + }; + } + ``` #### Directives 1. Shorthand `@` is preferable over `v-on` - ```javascript - // bad - <component v-on:click="eventHandler"/> + ```javascript + // bad + <component v-on:click="eventHandler"/> - // good - <component @click="eventHandler"/> - ``` + // good + <component @click="eventHandler"/> + ``` -2. Shorthand `:` is preferable over `v-bind` +1. Shorthand `:` is preferable over `v-bind` - ```javascript - // bad - <component v-bind:class="btn"/> + ```javascript + // bad + <component v-bind:class="btn"/> - // good - <component :class="btn"/> - ``` + // good + <component :class="btn"/> + ``` -3. Shorthand `#` is preferable over `v-slot` +1. Shorthand `#` is preferable over `v-slot` - ```javascript - // bad - <template v-slot:header></template> + ```javascript + // bad + <template v-slot:header></template> - // good - <template #header></template> - ``` + // good + <template #header></template> + ``` #### Closing tags 1. Prefer self closing component tags - ```javascript - // bad - <component></component> + ```javascript + // bad + <component></component> - // good - <component /> - ``` + // good + <component /> + ``` #### Ordering 1. Tag order in `.vue` file - ``` - <script> - // ... - </script> - - <template> - // ... - </template> - - // We don't use scoped styles but there are few instances of this - <style> - // ... - </style> - ``` + ``` + <script> + // ... + </script> + + <template> + // ... + </template> + + // We don't use scoped styles but there are few instances of this + <style> + // ... + </style> + ``` 1. Properties in a Vue Component: Check [order of properties in components rule][vue-order]. @@ -608,50 +609,50 @@ When using `v-for` you need to provide a *unique* `:key` attribute for each item 1. If the elements of the array being iterated have an unique `id` it is advised to use it: - ```html - <div - v-for="item in items" - :key="item.id" - > - <!-- content --> - </div> - ``` + ```html + <div + v-for="item in items" + :key="item.id" + > + <!-- content --> + </div> + ``` 1. When the elements being iterated don't have a unique id, you can use the array index as the `:key` attribute - ```html - <div - v-for="(item, index) in items" - :key="index" - > - <!-- content --> - </div> - ``` + ```html + <div + v-for="(item, index) in items" + :key="index" + > + <!-- content --> + </div> + ``` 1. When using `v-for` with `template` and there is more than one child element, the `:key` values must be unique. It's advised to use `kebab-case` namespaces. - ```html - <template v-for="(item, index) in items"> - <span :key="`span-${index}`"></span> - <button :key="`button-${index}`"></button> - </template> - ``` + ```html + <template v-for="(item, index) in items"> + <span :key="`span-${index}`"></span> + <button :key="`button-${index}`"></button> + </template> + ``` 1. When dealing with nested `v-for` use the same guidelines as above. - ```html - <div - v-for="item in items" - :key="item.id" - > - <span - v-for="element in array" - :key="element.id" - > - <!-- content --> - </span> - </div> - ``` + ```html + <div + v-for="item in items" + :key="item.id" + > + <span + v-for="element in array" + :key="element.id" + > + <!-- content --> + </span> + </div> + ``` Useful links: @@ -662,35 +663,35 @@ Useful links: 1. Tooltips: Do not rely on `has-tooltip` class name for Vue components - ```javascript - // bad - <span - class="has-tooltip" - title="Some tooltip text"> - Text - </span> - - // good - <span - v-tooltip - title="Some tooltip text"> - Text - </span> - ``` + ```javascript + // bad + <span + class="has-tooltip" + title="Some tooltip text"> + Text + </span> + + // good + <span + v-tooltip + title="Some tooltip text"> + Text + </span> + ``` 1. Tooltips: When using a tooltip, include the tooltip directive, `./app/assets/javascripts/vue_shared/directives/tooltip.js` 1. Don't change `data-original-title`. - ```javascript - // bad - <span data-original-title="tooltip text">Foo</span> + ```javascript + // bad + <span data-original-title="tooltip text">Foo</span> - // good - <span title="tooltip text">Foo</span> + // good + <span title="tooltip text">Foo</span> - $('span').tooltip('_fixTitle'); - ``` + $('span').tooltip('_fixTitle'); + ``` ### The Javascript/Vue Accord @@ -713,8 +714,6 @@ The goal of this accord is to make sure we are all on the same page. [airbnb-js-style-guide]: https://github.com/airbnb/javascript [eslintrc]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.eslintrc -[eslint-this]: http://eslint.org/docs/rules/class-methods-use-this -[eslint-new]: http://eslint.org/docs/rules/no-new [eslint-plugin-vue]: https://github.com/vuejs/eslint-plugin-vue [eslint-plugin-vue-rules]: https://github.com/vuejs/eslint-plugin-vue#bulb-rules [vue-order]: https://github.com/vuejs/eslint-plugin-vue/blob/master/docs/rules/order-in-components.md diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index 5220c9eeea3..95c4a094c04 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -35,7 +35,7 @@ New utility classes should be added to [`utilities.scss`](https://gitlab.com/git We recommend a "utility-first" approach. 1. Start with utility classes. -2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it. +1. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it. This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). @@ -212,6 +212,7 @@ selectors are intended for use only with JavaScript to allow for removal or renaming without breaking styling. ### IDs + Don't use ID selectors in CSS. ```scss diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 6c7572352ec..421b7265613 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -34,6 +34,7 @@ new_feature │ └── new_feature_store.js ├── index.js ``` + _For consistency purposes, we recommend you to follow the same structure._ Let's look into each of them: diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index bf248b7f8af..557d3132d71 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -1,15 +1,18 @@ # Vuex + To manage the state of an application you should use [Vuex][vuex-docs]. _Note:_ All of the below is explained in more detail in the official [Vuex documentation][vuex-docs]. ## Separation of concerns + Vuex is composed of State, Getters, Mutations, Actions and Modules. When a user clicks on an action, we need to `dispatch` it. This action will `commit` a mutation that will change the state. _Note:_ The action itself will not update the state, only a mutation should update the state. ## File structure + When using Vuex at GitLab, separate this concerns into different files to improve readability: ``` @@ -21,10 +24,12 @@ When using Vuex at GitLab, separate this concerns into different files to improv ├── state.js # state └── mutation_types.js # mutation types ``` + The following example shows an application that lists and adds users to the state. (For a more complex example implementation take a look at the security applications store in [here](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)) ### `index.js` + This is the entry point for our store. You can use the following as a guide: ```javascript @@ -47,6 +52,7 @@ export default createStore(); ``` ### `state.js` + The first thing you should do before writing any code is to design the state. Often we need to provide data from haml to our Vue application. Let's store it in the state for better access. @@ -66,9 +72,11 @@ Often we need to provide data from haml to our Vue application. Let's store it i ``` #### Access `state` properties + You can use `mapState` to access state properties in the components. ### `actions.js` + An action is a payload of information to send data from our application to our store. An action is usually composed by a `type` and a `payload` and they describe what happened. @@ -110,6 +118,7 @@ In this file, we will write the actions that will call the respective mutations: ``` #### Actions Pattern: `request` and `receive` namespaces + When a request is made we often want to show a loading state to the user. Instead of creating an action to toggle the loading state and dispatch it in the component, @@ -136,6 +145,7 @@ By following this pattern we guarantee: 1. Actions are simple and straightforward #### Dispatching actions + To dispatch an action from a component, use the `mapActions` helper: ```javascript @@ -154,6 +164,7 @@ import { mapActions } from 'vuex'; ``` ### `mutations.js` + The mutations specify how the application state changes in response to actions sent to the store. The only way to change state in a Vuex store should be by committing a mutation. @@ -193,6 +204,7 @@ Remember that actions only describe that something happened, they don't describe ``` ### `getters.js` + Sometimes we may need to get derived state based on store state, like filtering for a specific prop. Using a getter will also cache the result based on dependencies due to [how computed props work](https://vuejs.org/v2/guide/computed.html#Computed-Caching-vs-Methods) This can be done through the `getters`: @@ -219,6 +231,7 @@ import { mapGetters } from 'vuex'; ``` ### `mutation_types.js` + From [vuex mutations docs][vuex-mutations]: > It is a commonly seen pattern to use constants for mutation types in various Flux implementations. This allows the code to take advantage of tooling like linters, and putting all constants in a single file allows your collaborators to get an at-a-glance view of what mutations are possible in the entire application. @@ -227,6 +240,7 @@ export const ADD_USER = 'ADD_USER'; ``` ### How to include the store in your application + The store should be included in the main component of your application: ```javascript @@ -241,6 +255,7 @@ The store should be included in the main component of your application: ``` ### Communicating with the Store + ```javascript <script> import { mapActions, mapState, mapGetters } from 'vuex'; @@ -298,29 +313,33 @@ export default { 1. Do not call a mutation directly. Always use an action to commit a mutation. Doing so will keep consistency throughout the application. From Vuex docs: - > why don't we just call store.commit('action') directly? Well, remember that mutations must be synchronous? Actions aren't. We can perform asynchronous operations inside an action. + > Why don't we just call store.commit('action') directly? Well, remember that mutations must be synchronous? Actions aren't. We can perform asynchronous operations inside an action. - ```javascript - // component.vue + ```javascript + // component.vue - // bad - created() { - this.$store.commit('mutation'); - } + // bad + created() { + this.$store.commit('mutation'); + } + + // good + created() { + this.$store.dispatch('action'); + } + ``` - // good - created() { - this.$store.dispatch('action'); - } - ``` 1. Use mutation types instead of hardcoding strings. It will be less error prone. 1. The State will be accessible in all components descending from the use where the store is instantiated. ### Testing Vuex + #### Testing Vuex concerns + Refer to [vuex docs][vuex-testing] regarding testing Actions, Getters and Mutations. #### Testing components that need a store + Smaller components might use `store` properties to access the data. In order to write unit tests for those components, we need to include the store and provide the correct state: @@ -363,6 +382,7 @@ describe('component', () => { ``` #### Testing Vuex actions and getters + Because we're currently using [`babel-plugin-rewire`](https://github.com/speedskater/babel-plugin-rewire), you may encounter the following error when testing your Vuex actions and getters: `[vuex] actions should be function or object with "handler" function` diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 56872f8c075..f1374b9e280 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -8,5 +8,5 @@ disable those changes, without having to revert an entire release. Before using feature flags for GitLab's development, read through the following: - [Process for using features flags](process.md). -- [Developing with feature flags documentation](development.md). -- [Controlling feature flags documentation](controls.md). +- [Developing with feature flags](development.md). +- [Controlling feature flags](controls.md). diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 02874d18a30..44af2b020a4 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -2,6 +2,8 @@ We use the [CarrierWave] gem to handle file upload, store and retrieval. +File uploads should be accelerated by workhorse, for details please refer to [uploads development documentation](uploads.md). + There are many places where file uploading is used, according to contexts: - System @@ -92,8 +94,8 @@ in your uploader, you need to either 1) include `RecordsUpload::Concern` and pre The `CarrierWave::Uploader#store_dir` is overridden to - - `GitlabUploader.base_dir` + `GitlabUploader.dynamic_segment` when the store is LOCAL - - `GitlabUploader.dynamic_segment` when the store is REMOTE (the bucket name is used to namespace) +- `GitlabUploader.base_dir` + `GitlabUploader.dynamic_segment` when the store is LOCAL +- `GitlabUploader.dynamic_segment` when the store is REMOTE (the bucket name is used to namespace) ### Using `ObjectStorage::Extension::RecordsUploads` diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md new file mode 100644 index 00000000000..dd8944ff1c8 --- /dev/null +++ b/doc/development/filtering_by_label.md @@ -0,0 +1,162 @@ +# Filtering by label + +## Introduction + +GitLab has [labels](../user/project/labels.md) that can be assigned to issues, +merge requests, and epics. Labels on those objects are a many-to-many relation +through the polymorphic `label_links` table. + +To filter these objects by multiple labels - for instance, 'all open +issues with the label ~Plan and the label ~backend' - we generate a +query containing a `GROUP BY` clause. In a simple form, this looks like: + +```sql +SELECT + issues.* +FROM + issues + INNER JOIN label_links ON label_links.target_id = issues.id + AND label_links.target_type = 'Issue' + INNER JOIN labels ON labels.id = label_links.label_id +WHERE + issues.project_id = 13083 + AND (issues.state IN ('opened')) + AND labels.title IN ('Plan', + 'backend') +GROUP BY + issues.id +HAVING (COUNT(DISTINCT labels.title) = 2) +ORDER BY + issues.updated_at DESC, + issues.id DESC +LIMIT 20 OFFSET 0 +``` + +In particular, note that: + +1. We `GROUP BY issues.id` so that we can ... +1. Use the `HAVING (COUNT(DISTINCT labels.title) = 2)` condition to ensure that + all matched issues have both labels. + +This is more complicated than is ideal. It makes the query construction more +prone to errors (such as +[issue #15557](https://gitlab.com/gitlab-org/gitlab-ce/issues/15557)). + +## Attempt A: WHERE EXISTS + +### Attempt A1: use multiple subqueries with WHERE EXISTS + +In [issue #37137](https://gitlab.com/gitlab-org/gitlab-ce/issues/37137) +and its associated [merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14022), +we tried to replace the `GROUP BY` with multiple uses of `WHERE EXISTS`. For the +example above, this would give: + +```sql +WHERE (EXISTS ( + SELECT + TRUE + FROM + label_links + INNER JOIN labels ON labels.id = label_links.label_id + WHERE + labels.title = 'Plan' + AND target_type = 'Issue' + AND target_id = issues.id)) +AND (EXISTS ( + SELECT + TRUE + FROM + label_links + INNER JOIN labels ON labels.id = label_links.label_id + WHERE + labels.title = 'backend' + AND target_type = 'Issue' + AND target_id = issues.id)) +``` + +While this worked without schema changes, and did improve readability somewhat, +it did not improve query performance. + +## Attempt B: Denormalize using an array column + +Having [removed MySQL support in GitLab 12.1](https://about.gitlab.com/2019/06/27/removing-mysql-support/), +using [Postgres's arrays](https://www.postgresql.org/docs/9.6/arrays.html) became more +tractable as we didn't have to support two databases. We discussed denormalizing +the `label_links` table for querying in +[issue #49651](https://gitlab.com/gitlab-org/gitlab-ce/issues/49651), +with two options: label IDs and titles. + +We can think of both of those as array columns on `issues`, `merge_requests`, +and `epics`: `issues.label_ids` would be an array column of label IDs, and +`issues.label_titles` would be an array of label titles. + +These array columns can be complemented with [GIN +indexes](https://www.postgresql.org/docs/9.6/gin-intro.html) to improve +matching. + +### Attempt B1: store label IDs for each object + +This has some strong advantages over titles: + +1. Unless a label is deleted, or a project is moved, we never need to + bulk-update the denormalized column. +1. It uses less storage than the titles. + +Unfortunately, our application design makes this hard. If we were able to query +just by label ID easily, we wouldn't need the `INNER JOIN labels` in the initial +query at the start of this document. GitLab allows users to filter by label +title across projects and even across groups, so a filter by the label ~Plan may +include labels with multiple distinct IDs. + +We do not want users to have to know about the different IDs, which means that +given this data set: + +| Project | ~Plan label ID | ~backend label ID | +| ------- | -------------- | ----------------- | +| A | 11 | 12 | +| B | 21 | 22 | +| C | 31 | 32 | + +We would need something like: + +```sql +WHERE + label_ids @> ARRAY[11, 12] + OR label_ids @> ARRAY[21, 22] + OR label_ids @> ARRAY[31, 32] +``` + +This can get even more complicated when we consider that in some cases, there +might be two ~backend labels - with different IDs - that could apply to the same +object, so the number of combinations would balloon further. + +### Attempt B2: store label titles for each object + +From the perspective of updating the labelable object, this is the worst +option. We have to bulk update the objects when: + +1. The objects are moved from one project to another. +1. The project is moved from one group to another. +1. The label is renamed. +1. The label is deleted. + +It also uses much more storage. Querying is simple, though: + +```sql +WHERE + label_titles @> ARRAY['Plan', 'backend'] +``` + +And our [tests in issue #49651](https://gitlab.com/gitlab-org/gitlab-ce/issues/49651#note_188777346) +showed that this could be fast. + +However, at present, the disadvantages outweigh the advantages. + +## Conclusion + +We have yet to find a method that is demonstratably better than the current +method, when considering: + +1. Query performance. +1. Readability. +1. Ease of maintaining schema consistency. diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index ec9718cea71..8d93c52e7bc 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -3,9 +3,9 @@ When adding a new entry to `Gemfile` or upgrading an existing dependency pay attention to the following rules. -## No gems fetched from git repositories +## No gems fetched from Git repositories -We do not allow gems that are fetched from git repositories. All gems have +We do not allow gems that are fetched from Git repositories. All gems have to be available in the RubyGems index. We want to minimize external build dependencies and build times. diff --git a/doc/development/geo.md b/doc/development/geo.md index 685d4e44ad3..cc3e2d1ccc5 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -170,7 +170,7 @@ while `pull` requests will continue to be served by the **secondary** node for m HTTPS and SSH requests are handled differently: - With HTTPS, we will give the user a `HTTP 302 Redirect` pointing to the project on the **primary** node. - The git client is wise enough to understand that status code and process the redirection. + The Git client is wise enough to understand that status code and process the redirection. - With SSH, because there is no equivalent way to perform a redirect, we have to proxy the request. This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request to the HTTP protocol, and then proxying it to the **primary** node. @@ -341,9 +341,9 @@ not used, so sessions etc. aren't shared between nodes. GitLab can optionally use Object Storage to store data it would otherwise store on disk. These things can be: - - LFS Objects - - CI Job Artifacts - - Uploads +- LFS Objects +- CI Job Artifacts +- Uploads Objects that are stored in object storage, are not handled by Geo. Geo ignores items in object storage. Either: @@ -412,15 +412,15 @@ The Geo **primary** stores events in the `geo_event_log` table. Each entry in the log contains a specific type of event. These type of events include: - - Repository Deleted event - - Repository Renamed event - - Repositories Changed event - - Repository Created event - - Hashed Storage Migrated event - - Lfs Object Deleted event - - Hashed Storage Attachments event - - Job Artifact Deleted event - - Upload Deleted event +- Repository Deleted event +- Repository Renamed event +- Repositories Changed event +- Repository Created event +- Hashed Storage Migrated event +- Lfs Object Deleted event +- Hashed Storage Attachments event +- Job Artifact Deleted event +- Upload Deleted event ### Geo Log Cursor @@ -526,4 +526,4 @@ old method: - Replication is synchronous and we preserve the order of events. - Replication of the events happen at the same time as the changes in the - database. + database. diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index c103a4527ff..4dd1edf9b5a 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -8,30 +8,6 @@ storage disk use. To counteract this problem, we are adding Git object deduplication for forks to GitLab. In this document, we will describe how GitLab implements Git object deduplication. -## Enabling Git object deduplication via feature flags - -As of GitLab 12.0, Git object deduplication in GitLab is still behind a -feature flag. In this document, you can read about the effects of -enabling the feature. Also, note that Git object deduplication is -limited to forks of public projects on hashed repository storage. - -You can enable deduplication globally by setting the `object_pools` -feature flag to `true`: - -``` {.ruby} -Feature.enable(:object_pools) -``` - -Or just for forks of a specific project: - -``` {.ruby} -fork_parent = Project.find(MY_PROJECT_ID) -Feature.enable(:object_pools, fork_parent) -``` - -To check if a project uses Git object deduplication, look in a Rails -console if `project.pool_repository` is present. - ## Pool repositories ### Understanding Git alternates @@ -79,11 +55,11 @@ at the Rails application level in SQL. In conclusion, we need three things for effective object deduplication across a collection of GitLab project repositories at the Git level: -1. A pool repository must exist. -2. The participating project repositories must be linked to the pool - repository via their respective `objects/info/alternates` files. -3. The pool repository must contain Git object data common to the - participating project repositories. +1. A pool repository must exist. +1. The participating project repositories must be linked to the pool + repository via their respective `objects/info/alternates` files. +1. The pool repository must contain Git object data common to the + participating project repositories. ### Deduplication factor @@ -105,71 +81,71 @@ With pool repositories we made a fresh start. These live in their own `pool_repositories` SQL table. The relations between these two tables are as follows: -- a `Project` belongs to at most one `PoolRepository` - (`project.pool_repository`) -- as an automatic consequence of the above, a `PoolRepository` has - many `Project`s -- a `PoolRepository` has exactly one "source `Project`" - (`pool.source_project`) +- a `Project` belongs to at most one `PoolRepository` + (`project.pool_repository`) +- as an automatic consequence of the above, a `PoolRepository` has + many `Project`s +- a `PoolRepository` has exactly one "source `Project`" + (`pool.source_project`) > TODO Fix invalid SQL data for pools created prior to GitLab 11.11 > <https://gitlab.com/gitlab-org/gitaly/issues/1653>. ### Assumptions -- All repositories in a pool must use [hashed - storage](../administration/repository_storage_types.md). This is so - that we don't have to ever worry about updating paths in - `object/info/alternates` files. -- All repositories in a pool must be on the same Gitaly storage shard. - The Git alternates mechanism relies on direct disk access across - multiple repositories, and we can only assume direct disk access to - be possible within a Gitaly storage shard. -- The only two ways to remove a member project from a pool are (1) to - delete the project or (2) to move the project to another Gitaly - storage shard. +- All repositories in a pool must use [hashed + storage](../administration/repository_storage_types.md). This is so + that we don't have to ever worry about updating paths in + `object/info/alternates` files. +- All repositories in a pool must be on the same Gitaly storage shard. + The Git alternates mechanism relies on direct disk access across + multiple repositories, and we can only assume direct disk access to + be possible within a Gitaly storage shard. +- The only two ways to remove a member project from a pool are (1) to + delete the project or (2) to move the project to another Gitaly + storage shard. ### Creating pools and pool memberships -- When a pool gets created, it must have a source project. The initial - contents of the pool repository are a Git clone of the source - project repository. -- The occasion for creating a pool is when an existing eligible - (public, hashed storage, non-forked) GitLab project gets forked and - this project does not belong to a pool repository yet. The fork - parent project becomes the source project of the new pool, and both - the fork parent and the fork child project become members of the new - pool. -- Once project A has become the source project of a pool, all future - eligible forks of A will become pool members. -- If the fork source is itself a fork, the resulting repository will - neither join the repository nor will a new pool repository be - seeded. - - eg: - - Suppose fork A is part of a pool repository, any forks created off - of fork A *will not* be a part of the pool repository that fork A is - a part of. - - Suppose B is a fork of A, and A does not belong to an object pool. - Now C gets created as a fork of B. C will not be part of a pool - repository. +- When a pool gets created, it must have a source project. The initial + contents of the pool repository are a Git clone of the source + project repository. +- The occasion for creating a pool is when an existing eligible + (public, hashed storage, non-forked) GitLab project gets forked and + this project does not belong to a pool repository yet. The fork + parent project becomes the source project of the new pool, and both + the fork parent and the fork child project become members of the new + pool. +- Once project A has become the source project of a pool, all future + eligible forks of A will become pool members. +- If the fork source is itself a fork, the resulting repository will + neither join the repository nor will a new pool repository be + seeded. + + eg: + + Suppose fork A is part of a pool repository, any forks created off + of fork A *will not* be a part of the pool repository that fork A is + a part of. + + Suppose B is a fork of A, and A does not belong to an object pool. + Now C gets created as a fork of B. C will not be part of a pool + repository. > TODO should forks of forks be deduplicated? > <https://gitlab.com/gitlab-org/gitaly/issues/1532> ### Consequences -- If a normal Project participating in a pool gets moved to another - Gitaly storage shard, its "belongs to PoolRepository" relation will - be broken. Because of the way moving repositories between shard is - implemented, we will automatically get a fresh self-contained copy - of the project's repository on the new storage shard. -- If the source project of a pool gets moved to another Gitaly storage - shard or is deleted the "source project" relation is not broken. - However, as of GitLab 12.0 a pool will not fetch from a source - unless the source is on the same Gitaly shard. +- If a normal Project participating in a pool gets moved to another + Gitaly storage shard, its "belongs to PoolRepository" relation will + be broken. Because of the way moving repositories between shard is + implemented, we will automatically get a fresh self-contained copy + of the project's repository on the new storage shard. +- If the source project of a pool gets moved to another Gitaly storage + shard or is deleted the "source project" relation is not broken. + However, as of GitLab 12.0 a pool will not fetch from a source + unless the source is on the same Gitaly shard. ## Consistency between the SQL pool relation and Gitaly @@ -193,7 +169,7 @@ There are three different things that can go wrong here. In this case, we miss out on disk space savings but all RPC's on A itself will function fine. The next time garbage collection runs on A, the alternates connection gets established in Gitaly. This is done by -`Projects::GitDeduplicationService` in gitlab-rails. +`Projects::GitDeduplicationService` in GitLab Rails. #### 2. SQL says repo A belongs to pool P1 but Gitaly says A has alternate objects in pool P2 diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 5552d5d37b4..592fc13873b 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -45,13 +45,13 @@ The process for adding new Gitaly features is: - release a new version of gitaly-proto - write implementation and tests for the RPC [in Gitaly](https://gitlab.com/gitlab-org/gitaly), in Go or Ruby - release a new version of Gitaly -- write client code in gitlab-ce/ee, gitlab-workhorse or gitlab-shell that calls the new Gitaly RPC +- write client code in GitLab CE/EE, GitLab Workhorse or GitLab Shell that calls the new Gitaly RPC These steps often overlap. It is possible to use an unreleased version of Gitaly and gitaly-proto during testing and development. - See the [Gitaly repo](https://gitlab.com/gitlab-org/gitaly/blob/master/CONTRIBUTING.md#development-and-testing-with-a-custom-gitaly-proto) for instructions on writing server side code with an unreleased protocol. -- See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running gitlab-ce tests with a modified version of Gitaly. +- See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running GitLab CE tests with a modified version of Gitaly. - In GDK run `gdk install` and restart `gdk run` (or `gdk run app`) to use a locally modified Gitaly version for development ### Gitaly-ruby @@ -146,7 +146,7 @@ Once the code is wrapped in this block, this code-path will be excluded from n+1 ## Request counts -Commits and other git data, is now fetched through Gitaly. These fetches can, +Commits and other Git data, is now fetched through Gitaly. These fetches can, much like with a database, be batched. This improves performance for the client and for Gitaly itself and therefore for the users too. To keep performance stable and guard performance regressions, Gitaly calls can be counted and the call count @@ -164,10 +164,10 @@ end ## Running tests with a locally modified version of Gitaly -Normally, gitlab-ce/ee tests use a local clone of Gitaly in +Normally, GitLab CE/EE tests use a local clone of Gitaly in `tmp/tests/gitaly` pinned at the version specified in `GITALY_SERVER_VERSION`. The `GITALY_SERVER_VERSION` file supports -`=my-branch` syntax to use a custom branch in gitlab-org/gitaly. If +`=my-branch` syntax to use a custom branch in <https://gitlab.com/gitlab-org/gitaly>. If you want to run tests locally against a modified version of Gitaly you can replace `tmp/tests/gitaly` with a symlink. This is much faster because the `=my-branch` syntax forces a Gitaly re-install each time @@ -237,24 +237,23 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. 1. Create prometheus metrics: ```go - var findAllTagsRequests = prometheus.NewCounterVec( - prometheus.CounterOpts{ - Name: "gitaly_find_all_tags_requests_total", - Help: "Counter of go vs ruby implementation of FindAllTags", - }, - []string{"implementation"}, - ) + var findAllTagsRequests = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "gitaly_find_all_tags_requests_total", + Help: "Counter of go vs ruby implementation of FindAllTags", + }, + []string{"implementation"}, ) func init() { - prometheus.Register(findAllTagsRequests) + prometheus.Register(findAllTagsRequests) } if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { - findAllTagsRequests.WithLabelValues("go").Inc() + findAllTagsRequests.WithLabelValues("go").Inc() // go implementation } else { - findAllTagsRequests.WithLabelValues("ruby").Inc() + findAllTagsRequests.WithLabelValues("ruby").Inc() // ruby implementation } ``` @@ -277,9 +276,9 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. require.NoError(t, err) ``` -### Gitlab-Rails +### GitLab Rails -1. Add feature flag to `lib/gitlab/gitaly_client.rb` (in gitlab-rails): +1. Add feature flag to `lib/gitlab/gitaly_client.rb` (in GitLab Rails): ```ruby SERVER_FEATURE_FLAGS = %w[go-find-all-tags].freeze diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index f09339eb3a4..83444093f9c 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -107,6 +107,32 @@ Modules](https://github.com/golang/go/wiki/Modules). It provides a way to define and lock dependencies for reproducible builds. It should be used whenever possible. +When Go Modules are in use, there should not be a `vendor/` directory. Instead, +Go will automatically download dependencies when they are needed to build the +project. This is in line with how dependencies are handled with Bundler in Ruby +projects, and makes merge requests easier to review. + +In some cases, such as building a Go project for it to act as a dependency of a +CI run for another project, removing the `vendor/` directory means the code must +be downloaded repeatedly, which can lead to intermittent problems due to rate +limiting or network failures. In these circumstances, you should cache the +downloaded code between runs with a `.gitlab-ci.yml` snippet like this: + +```yaml +.go-cache: + variables: + GOPATH: $CI_PROJECT_DIR/.go + before_script: + - mkdir -p .go + cache: + paths: + - .go/pkg/mod/ + +test: + extends: .go-cache + # ... +``` + There was a [bug on modules checksums](https://github.com/golang/go/issues/29278) in Go < v1.11.4, so make sure to use at least this version to avoid `checksum mismatch` errors. @@ -129,17 +155,89 @@ deploy a new pod, migrating the data automatically. ## Testing +### Testing frameworks + We should not use any specific library or framework for testing, as the [standard library](https://golang.org/pkg/) provides already everything to get -started. For example, some external dependencies might be worth considering in -case we decide to use a specific library or framework: +started. If there is a need for more sophisticated testing tools, the following +external dependencies might be worth considering in case we decide to use a specific +library or framework: - [Testify](https://github.com/stretchr/testify) - [httpexpect](https://github.com/gavv/httpexpect) +### Subtests + Use [subtests](https://blog.golang.org/subtests) whenever possible to improve code readability and test output. +### Better output in tests + +When comparing expected and actual values in tests, use +[testify/require.Equal](https://godoc.org/github.com/stretchr/testify/require#Equal), +[testify/require.EqualError](https://godoc.org/github.com/stretchr/testify/require#EqualError), +[testify/require.EqualValues](https://godoc.org/github.com/stretchr/testify/require#EqualValues), +and others to improve readability when comparing structs, errors, +large portions of text, or JSON documents: + +```go +type TestData struct { + // ... +} + +func FuncUnderTest() TestData { + // ... +} + +func Test(t *testing.T) { + t.Run("FuncUnderTest", func(t *testing.T) { + want := TestData{} + got := FuncUnderTest() + + require.Equal(t, want, got) // note that expected value comes first, then comes the actual one ("diff" semantics) + }) +} +``` + +### Table-Driven Tests + +Using [Table-Driven Tests](https://github.com/golang/go/wiki/TableDrivenTests) +is generally good practice when you have multiple entries of +inputs/outputs for the same function. Below are some guidelines one can +follow when writing table-driven test. These guidelines are mostly +extracted from Go standard library source code. Keep in mind it's OK not +to follow these guidelines when it makes sense. + +#### Defining test cases + +Each table entry is a complete test case with inputs and expected +results, and sometimes with additional information such as a test name +to make the test output easily readable. + +- [Define a slice of anonymous struct](https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/csv/reader_test.go#L16) + inside of the test. +- [Define a slice of anonymous struct](https://github.com/golang/go/blob/55d31e16c12c38d36811bdee65ac1f7772148250/src/cmd/go/internal/module/module_test.go#L9-L66) + outside of the test. +- [Named structs](https://github.com/golang/go/blob/2e0cd2aef5924e48e1ceb74e3d52e76c56dd34cc/src/cmd/go/internal/modfetch/coderepo_test.go#L54-L69) + for code reuse. +- [Using `map[string]struct{}`](https://github.com/golang/go/blob/6d5caf38e37bf9aeba3291f1f0b0081f934b1187/src/cmd/trace/annotations_test.go#L180-L235). + +#### Contents of the test case + +- Ideally, each test case should have a field with a unique identifier + to use for naming subtests. In the Go standard library, this is commonly the + `name string` field. +- Use `want`/`expect`/`actual` when you are specifcing something in the + test case that will be used for assertion. + +#### Variable names + +- Each table-driven test map/slice of struct can be named `tests`. +- When looping through `tests` the anonymous struct can be referred + to as `tt` or `tc`. +- The description of the test can be referred to as + `name`/`testName`/`tn`. + ### Benchmarks Programs handling a lot of IO or complex operations should always include diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index 13dda17bb7d..941eea2609e 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -53,7 +53,7 @@ When run, this spec doesn't do what we might expect: (compared using ==) ``` -That's because FactoryBot sequences are not reseted for each example. +This is because FactoryBot sequences are not reset for each example. Please remember that sequence-generated values exist only to avoid having to explicitly set attributes that have a uniqueness constraint when using a factory. diff --git a/doc/development/hash_indexes.md b/doc/development/hash_indexes.md index e6c1b3590b1..417ea18e22f 100644 --- a/doc/development/hash_indexes.md +++ b/doc/development/hash_indexes.md @@ -1,6 +1,6 @@ # Hash Indexes -Both PostgreSQL and MySQL support hash indexes besides the regular btree +PostgreSQL supports hash indexes besides the regular btree indexes. Hash indexes however are to be avoided at all costs. While they may _sometimes_ provide better performance the cost of rehashing can be very high. More importantly: at least until PostgreSQL 10.0 hash indexes are not diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 17462887162..141f5a8d6d9 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -21,18 +21,18 @@ The following tools are used: 1. [`gettext_i18n_rails`](https://github.com/grosser/gettext_i18n_rails): this gem allow us to translate content from models, views and controllers. Also it gives us access to the following raketasks: - - `rake gettext:find`: Parses almost all the files from the - Rails application looking for content that has been marked for - translation. Finally, it updates the PO files with the new content that - it has found. - - `rake gettext:pack`: Processes the PO files and generates the - MO files that are binary and are finally used by the application. + - `rake gettext:find`: Parses almost all the files from the + Rails application looking for content that has been marked for + translation. Finally, it updates the PO files with the new content that + it has found. + - `rake gettext:pack`: Processes the PO files and generates the + MO files that are binary and are finally used by the application. 1. [`gettext_i18n_rails_js`](https://github.com/webhippie/gettext_i18n_rails_js): this gem is useful to make the translations available in JavaScript. It provides the following raketask: - - `rake gettext:po_to_json`: Reads the contents from the PO files and - generates JSON files containing all the available translations. + - `rake gettext:po_to_json`: Reads the contents from the PO files and + generates JSON files containing all the available translations. 1. PO editor: there are multiple applications that can help us to work with PO files, a good option is [Poedit](https://poedit.net/download) which is @@ -139,60 +139,61 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s - In Ruby/HAML: - ```ruby - _("Hello %{name}") % { name: 'Joe' } => 'Hello Joe' - ``` + ```ruby + _("Hello %{name}") % { name: 'Joe' } => 'Hello Joe' + ``` - In JavaScript: - ```js - import { __, sprintf } from '~/locale'; + ```js + import { __, sprintf } from '~/locale'; - sprintf(__('Hello %{username}'), { username: 'Joe' }); // => 'Hello Joe' - ``` + sprintf(__('Hello %{username}'), { username: 'Joe' }); // => 'Hello Joe' + ``` - By default, `sprintf` escapes the placeholder values. - If you want to take care of that yourself, you can pass `false` as third argument. + By default, `sprintf` escapes the placeholder values. + If you want to take care of that yourself, you can pass `false` as third argument. - ```js - import { __, sprintf } from '~/locale'; + ```js + import { __, sprintf } from '~/locale'; - sprintf(__('This is %{value}'), { value: '<strong>bold</strong>' }); // => 'This is <strong>bold</strong>' - sprintf(__('This is %{value}'), { value: '<strong>bold</strong>' }, false); // => 'This is <strong>bold</strong>' - ``` + sprintf(__('This is %{value}'), { value: '<strong>bold</strong>' }); // => 'This is <strong>bold</strong>' + sprintf(__('This is %{value}'), { value: '<strong>bold</strong>' }, false); // => 'This is <strong>bold</strong>' + ``` ### Plurals - In Ruby/HAML: - ```ruby - n_('Apple', 'Apples', 3) - # => 'Apples' - ``` + ```ruby + n_('Apple', 'Apples', 3) + # => 'Apples' + ``` - Using interpolation: - ```ruby - n_("There is a mouse.", "There are %d mice.", size) % size - # => When size == 1: 'There is a mouse.' - # => When size == 2: 'There are 2 mice.' - ``` + Using interpolation: - Avoid using `%d` or count variables in singular strings. This allows more natural translation in some languages. + ```ruby + n_("There is a mouse.", "There are %d mice.", size) % size + # => When size == 1: 'There is a mouse.' + # => When size == 2: 'There are 2 mice.' + ``` + + Avoid using `%d` or count variables in singular strings. This allows more natural translation in some languages. - In JavaScript: - ```js - n__('Apple', 'Apples', 3) - // => 'Apples' - ``` + ```js + n__('Apple', 'Apples', 3) + // => 'Apples' + ``` - Using interpolation: + Using interpolation: - ```js - n__('Last day', 'Last %d days', x) - // => When x == 1: 'Last day' - // => When x == 2: 'Last 2 days' - ``` + ```js + n__('Last day', 'Last %d days', x) + // => When x == 1: 'Last day' + // => When x == 2: 'Last 2 days' + ``` ### Namespaces @@ -202,17 +203,17 @@ Namespaces should be PascalCase. - In Ruby/HAML: - ```ruby - s_('OpenedNDaysAgo|Opened') - ``` + ```ruby + s_('OpenedNDaysAgo|Opened') + ``` - In case the translation is not found it will return `Opened`. + In case the translation is not found it will return `Opened`. - In JavaScript: - ```js - s__('OpenedNDaysAgo|Opened') - ``` + ```js + s__('OpenedNDaysAgo|Opened') + ``` Note: The namespace should be removed from the translation. See the [translation guidelines for more details](translation.md#namespaced-strings). @@ -235,12 +236,12 @@ This makes use of [`Intl.DateTimeFormat`]. - In Ruby/HAML, we have two ways of adding format to dates and times: 1. **Through the `l` helper**, i.e. `l(active_session.created_at, format: :short)`. We have some predefined formats for - [dates](https://gitlab.com/gitlab-org/gitlab-ce/blob/v11.7.0/config/locales/en.yml#L54) and [times](https://gitlab.com/gitlab-org/gitlab-ce/blob/v11.7.0/config/locales/en.yml#L261). - If you need to add a new format, because other parts of the code could benefit from it, - you'll need to add it to [en.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/locales/en.yml) file. + [dates](https://gitlab.com/gitlab-org/gitlab-ce/blob/v11.7.0/config/locales/en.yml#L54) and [times](https://gitlab.com/gitlab-org/gitlab-ce/blob/v11.7.0/config/locales/en.yml#L261). + If you need to add a new format, because other parts of the code could benefit from it, + you'll need to add it to [en.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/locales/en.yml) file. 1. **Through `strftime`**, i.e. `milestone.start_date.strftime('%b %-d')`. We use `strftime` in case none of the formats - defined on [en.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/locales/en.yml) matches the date/time - specifications we need, and if there is no need to add it as a new format because is very particular (i.e. it's only used in a single view). + defined on [en.yml](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/locales/en.yml) matches the date/time + specifications we need, and if there is no need to add it as a new format because is very particular (i.e. it's only used in a single view). ## Best practices @@ -268,40 +269,40 @@ should be externalized as follows: This also applies when using links in between translated sentences, otherwise these texts are not translatable in certain languages. - In Ruby/HAML, instead of: - - ```haml - - zones_link = link_to(s_('ClusterIntegration|zones'), 'https://cloud.google.com/compute/docs/regions-zones/regions-zones', target: '_blank', rel: 'noopener noreferrer') - = s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } - ``` - - Set the link starting and ending HTML fragments as variables like so: - - ```haml - - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' - - zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } - = s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } - ``` + + ```haml + - zones_link = link_to(s_('ClusterIntegration|zones'), 'https://cloud.google.com/compute/docs/regions-zones/regions-zones', target: '_blank', rel: 'noopener noreferrer') + = s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } + ``` + + Set the link starting and ending HTML fragments as variables like so: + + ```haml + - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' + - zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } + = s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } + ``` - In JavaScript, instead of: - ```js - {{ - sprintf(s__("ClusterIntegration|Learn more about %{link}"), { - link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' - }) - }} - ``` - - Set the link starting and ending HTML fragments as variables like so: - - ```js - {{ - sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { - linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">' - linkEnd: '</a>', - }) - }} - ``` + ```js + {{ + sprintf(s__("ClusterIntegration|Learn more about %{link}"), { + link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' + }) + }} + ``` + + Set the link starting and ending HTML fragments as variables like so: + + ```js + {{ + sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { + linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">' + linkEnd: '</a>', + }) + }} + ``` The reasoning behind this is that in some languages words change depending on context. For example in Japanese は is added to the subject of a sentence and を to the object. This is impossible to translate correctly if we extract individual words from the sentence. @@ -374,29 +375,29 @@ Let's suppose you want to add translations for a new language, let's say French. 1. The first step is to register the new language in `lib/gitlab/i18n.rb`: - ```ruby - ... - AVAILABLE_LANGUAGES = { - ..., - 'fr' => 'Français' - }.freeze - ... - ``` + ```ruby + ... + AVAILABLE_LANGUAGES = { + ..., + 'fr' => 'Français' + }.freeze + ... + ``` 1. Next, you need to add the language: - ```sh - bin/rake gettext:add_language[fr] - ``` + ```sh + bin/rake gettext:add_language[fr] + ``` - If you want to add a new language for a specific region, the command is similar, - you just need to separate the region with an underscore (`_`). For example: + If you want to add a new language for a specific region, the command is similar, + you just need to separate the region with an underscore (`_`). For example: - ```sh - bin/rake gettext:add_language[en_GB] - ``` + ```sh + bin/rake gettext:add_language[en_GB] + ``` - Please note that you need to specify the region part in capitals. + Please note that you need to specify the region part in capitals. 1. Now that the language is added, a new directory has been created under the path: `locale/fr/`. You can now start using your PO editor to edit the PO file @@ -406,9 +407,9 @@ Let's suppose you want to add translations for a new language, let's say French. in order to generate the binary MO files and finally update the JSON files containing the translations: - ```sh - bin/rake gettext:compile - ``` + ```sh + bin/rake gettext:compile + ``` 1. In order to see the translated content we need to change our preferred language which can be found under the user's **Settings** (`/profile`). @@ -416,7 +417,7 @@ Let's suppose you want to add translations for a new language, let's say French. 1. After checking that the changes are ok, you can proceed to commit the new files. For example: - ```sh - git add locale/fr/ app/assets/javascripts/locale/fr/ - git commit -m "Add French translations for Cycle Analytics page" - ``` + ```sh + git add locale/fr/ app/assets/javascripts/locale/fr/ + git commit -m "Add French translations for Cycle Analytics page" + ``` diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 35c5b155594..492e3d48164 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -80,6 +80,7 @@ are very appreciative of the work done by translators and proofreaders! - Russian - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) + - Mark Minakou - [GitLab](https://gitlab.com/sandzhaj), [Crowdin](https://crowdin.com/profile/sandzhaj) - NickVolynkin - [Crowdin](https://crowdin.com/profile/NickVolynkin) - Serbian (Cyrillic) - Proofreaders needed. @@ -106,32 +107,31 @@ are very appreciative of the work done by translators and proofreaders! 1. Contribute translations to GitLab. See instructions for [translating GitLab](translation.md). - Translating GitLab is a community effort that requires team work and - attention to detail. Proofreaders play an important role helping new - contributors, and ensuring the consistency and quality of translations. - Your conduct and contributions as a translator should reflect this before - requesting to be a proofreader. + Translating GitLab is a community effort that requires team work and + attention to detail. Proofreaders play an important role helping new + contributors, and ensuring the consistency and quality of translations. + Your conduct and contributions as a translator should reflect this before + requesting to be a proofreader. 1. Request proofreader permissions by opening a merge request to add yourself to the list of proofreaders. - Open the [proofreader.md source file][proofreader-src] and click **Edit**. + Open the [proofreader.md source file][proofreader-src] and click **Edit**. - Add your language in alphabetical order, and add yourself to the list - including: - - name - - link to your GitLab profile - - link to your CrowdIn profile + Add your language in alphabetical order, and add yourself to the list + including: + - name + - link to your GitLab profile + - link to your CrowdIn profile - In the merge request description, please include links to any projects you - have previously translated. + In the merge request description, please include links to any projects you + have previously translated. 1. Your request to become a proofreader will be considered on the merits of your previous translations by [GitLab team members](https://about.gitlab.com/team/) or [Core team members](https://about.gitlab.com/core-team/) who are fluent in the language or current proofreaders. - When a request is made for the first proofreader for a language and there are no [GitLab team members](https://about.gitlab.com/team/) - or [Core team members](https://about.gitlab.com/core-team/) who speak the language, we will request links to previous translation work in other communities or projects. - + or [Core team members](https://about.gitlab.com/core-team/) who speak the language, we will request links to previous translation work in other communities or projects. [proofreader-src]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/i18n/proofreader.md diff --git a/doc/development/img/architecture_simplified.png b/doc/development/img/architecture_simplified.png Binary files differindex 1698c167c5e..1ad57b65468 100644 --- a/doc/development/img/architecture_simplified.png +++ b/doc/development/img/architecture_simplified.png diff --git a/doc/development/img/distributed_tracing_jaeger_ui.png b/doc/development/img/distributed_tracing_jaeger_ui.png Binary files differindex 57517dacced..dcd18b1ec9f 100644 --- a/doc/development/img/distributed_tracing_jaeger_ui.png +++ b/doc/development/img/distributed_tracing_jaeger_ui.png diff --git a/doc/development/img/distributed_tracing_performance_bar.png b/doc/development/img/distributed_tracing_performance_bar.png Binary files differindex c9998cedd2d..8c819045104 100644 --- a/doc/development/img/distributed_tracing_performance_bar.png +++ b/doc/development/img/distributed_tracing_performance_bar.png diff --git a/doc/development/img/elasticsearch_architecture.svg b/doc/development/img/elasticsearch_architecture.svg new file mode 100644 index 00000000000..2f38f9b04ee --- /dev/null +++ b/doc/development/img/elasticsearch_architecture.svg @@ -0,0 +1 @@ +<svg version="1.2" width="210mm" height="297mm" viewBox="0 0 21000 29700" preserveAspectRatio="xMidYMid" fill-rule="evenodd" stroke-width="28.222" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><defs class="ClipPathGroup"><clipPath id="a" clipPathUnits="userSpaceOnUse"><path d="M0 0h21000v29700H0z"/></clipPath></defs><g class="SlideGroup"><g class="Slide" clip-path="url(#a)"><g class="Page"><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 5575h3051v1651H1975z"/><path fill="#FFF" d="M3500 7200H2000V5600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7200H2000V5600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="6311"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="6785"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1475 3975h4051v3551H1475z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7500H1500V4000h4000v3500H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="1788" y="5048"><tspan>ApplicationSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 4675h8051v701H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v-650h4000"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 5325h8051v1101H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v1050h4000"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 2875h4951v4951H1075z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3550 7800H1100V2900h4900v4900H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="1946" y="3514"><tspan fill="#C9211E">SnippetsSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 12175h3051v1651H1975z"/><path fill="#FFF" d="M3500 13800H2000v-1600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 13800H2000v-1600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="12911"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="13385"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 10775h4951v3251H1075z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000H1100v-3200h4900v3200H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2511" y="11461"><tspan>Application</tspan></tspan><tspan class="TextPosition" x="1933" y="11935"><tspan>VersionedSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v7451H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v7400h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 14075h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 14900h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="14648"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 13075h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 15200h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="13731"><tspan fill="#C9211E">V12p1::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 14575h3051v1851H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 16400H8000v-1800h3000v1800H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="15411"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8429" y="15885"><tspan>ClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 16875h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 17700h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="17448"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 15875h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 18000h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="16531"><tspan fill="#C9211E">V12p2::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 14125h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 15475h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v1450h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v1551H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v1500h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 19975h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 20800h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="20548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 18975h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 21100h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="19631"><tspan fill="#C9211E">V12p1::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 20275h3051v2251H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 22500H8000v-2200h3000v2200H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="21311"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8154" y="21785"><tspan>InstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 22775h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 23600h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="23348"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 21775h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 23900h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="22431"><tspan fill="#C9211E">V12p2::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 20025h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 21375h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v1450h937"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 1600h10697v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="2233"><tspan>Standard elasticsearch-rails setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 9300h7683v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="9933"><tspan>GitLab multi-indices setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3400 21300h4821v1197H3400z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4250" y="21840"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="3651" y="22264"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3380 15400h4821v1197H3380z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4512" y="15940"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="3631" y="16364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 3500h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="10132" y="4040"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="9251" y="4464"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 6400h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="9850" y="6940"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="9251" y="7364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 25175h2051v851H1975z"/><path fill="none" stroke="#999" stroke-width="50" d="M3000 26000H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2634" y="25748"><tspan fill="gray">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 25200h7101v726H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="25710"><tspan>elasticsearch-rails’ internal class</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 26400h8601v1200H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="26910"><tspan>where model-specific logic is</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 26275h2051v851H1975z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3000 27100H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="2613" y="26848"><tspan fill="#C9211E">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4900 17289h5901v2312H4900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="7236" y="17748"><tspan fill="gray">Write operations like </tspan></tspan><tspan class="TextPosition" x="5323" y="18159"><tspan fill="gray">indexing/updating are forwarded </tspan></tspan><tspan class="TextPosition" x="8024" y="18570"><tspan fill="gray">to all instances.</tspan></tspan></tspan><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="5501" y="18981"><tspan fill="gray">Read operations are forwarded </tspan></tspan><tspan class="TextPosition" x="7126" y="19392"><tspan fill="gray">to specified instance.</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 15769h1422v2691h-1422z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1429 0 934-1618 1119-2337"/><path fill="#999" d="M12206 15769l-460 293 267 217 193-510z"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 18429h1528v2862h-1528z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1509 0 970 1782 1200 2526"/><path fill="#999" d="M12312 21290l-227-496-252 235 479 261z"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M1800 24000h7101v807H1800z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="494" font-weight="700"><tspan class="TextPosition" x="2050" y="24574"><tspan>Legend</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 4275h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 5100h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14737" y="4848"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 5975h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 6800h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14462" y="6548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g></g></g></g></svg>
\ No newline at end of file diff --git a/doc/development/import_export.md b/doc/development/import_export.md index 64c91f151c5..0c343bc22e4 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -19,6 +19,7 @@ Project.find_by_full_path('group/project').import_state.slice(:jid, :status, :la grep JID /var/log/gitlab/sidekiq/current grep "Import/Export error" /var/log/gitlab/sidekiq/current grep "Import/Export backtrace" /var/log/gitlab/sidekiq/current +tail /var/log/gitlab/gitlab-rails/importer.log ``` ## Troubleshooting performance issues @@ -229,8 +230,8 @@ meaning that we want to bump the version up in the next version (or patch releas For example: 1. Add rename to `RelationRenameService` in X.Y -2. Remove it from `RelationRenameService` in X.Y + 1 -3. Bump Import/Export version in X.Y + 1 +1. Remove it from `RelationRenameService` in X.Y + 1 +1. Bump Import/Export version in X.Y + 1 ```ruby module Gitlab diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index 5f95cf3707c..777d372ec60 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -1,6 +1,6 @@ # Instrumenting Ruby Code -GitLab Performance Monitoring allows instrumenting of both methods and custom +[GitLab Performance Monitoring](../administration/monitoring/performance/index.md) allows instrumenting of both methods and custom blocks of Ruby code. Method instrumentation is the primary form of instrumentation with block-based instrumentation only being used when we want to drill down to specific regions of code within a method. diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index 9ba3b922fd8..e1350b02262 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -30,9 +30,11 @@ The following are required to install and test the app: 1. In the **From this URL** field, provide a link to the app descriptor. The host and port must point to your GitLab instance. For example: + ``` https://xxxx.serveo.net/-/jira_connect/app_descriptor.json ``` + 1. Click **Upload**. If the install was successful, you should see the **GitLab for Jira** app under **Manage apps**. diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md new file mode 100644 index 00000000000..5e6dc8d460a --- /dev/null +++ b/doc/development/interacting_components.md @@ -0,0 +1,29 @@ +# Developing against interacting components or features + +It's not uncommon that a single code change can reflect and interact with multiple parts of GitLab +codebase. Furthermore, an existing feature might have an underlying integration or behavior that +might go unnoticed even by reviewers and maintainers. + +The goal of this section is to briefly list interacting pieces to think about +when making _backend_ changes that might involve multiple features or [components](architecture.md#components). + +## Uploads + +GitLab supports uploads to [object storage]. That means every feature and +change that affects uploads should also be tested against [object storage], +which is _not_ enabled by default in [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit). + +When working on a related feature, make sure to enable and test it +against [Minio](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/object_storage.md). + +See also [File Storage in GitLab](file_storage.md). + +## Merge requests + +### Forks + +GitLab supports a great amount of features for [merge requests](../user/project/merge_requests/index.md). One +of them is the ability to create merge requests from and to [forks](../gitlab-basics/fork-project.md), +which should also be highly considered and tested upon development phase. + +[object storage]: https://docs.gitlab.com/charts/advanced/external-object-storage/ diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 4b2d48903ac..f4528667814 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -107,7 +107,7 @@ Mitigation strategies include: ## Debugging Logs related to the Kubernetes integration can be found in -[kubernetes.log](../administration/logs.md#kuberneteslog). On a local +[`kubernetes.log`](../administration/logs.md#kuberneteslog). On a local GDK install, this will be present in `log/kubernetes.log`. Some services such as diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 8c3408eb6e2..cb4c2d8967b 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -8,4 +8,4 @@ In April 2019, Francisco Javier López hosted a [Deep Dive] on GitLab's [Git LFS [Git LFS]: ../workflow/lfs/manage_large_binaries_with_git_lfs.html [recording on YouTube]: https://www.youtube.com/watch?v=Yyxwcksr0Qc [Google Slides]: https://docs.google.com/presentation/d/1E-aw6-z0rYd0346YhIWE7E9A65zISL9iIMAOq2zaw9E/edit -[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf
\ No newline at end of file +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 80ec7b8c0cf..29e4ace157b 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -14,7 +14,7 @@ it should be restricted on namespace scope. 1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in `ee/app/models/license.rb`. Note on `ee/app/models/ee/namespace.rb` that _Bronze_ GitLab.com features maps to on-premise _EES_, _Silver_ to _EEP_ and _Gold_ to _EEU_. -2. Check using: +1. Check using: ```ruby project.feature_available?(:feature_symbol) @@ -29,8 +29,8 @@ the instance license. 1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in `ee/app/models/license.rb`. -2. Add the same feature symbol to `GLOBAL_FEATURES` -3. Check using: +1. Add the same feature symbol to `GLOBAL_FEATURES` +1. Check using: ```ruby License.feature_available?(:feature_symbol) diff --git a/doc/development/logging.md b/doc/development/logging.md index 4f63c84fc0e..b43f1029cc6 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -133,7 +133,7 @@ importer progresses. Here's what to do: logs in `/var/log/gitlab/gitlab-rails/*.log` every hour and [keep at most 30 compressed files](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate). On GitLab.com, that setting is only 6 compressed files. These settings should suffice - for most users, but you may need to tweak them in [omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab). + for most users, but you may need to tweak them in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). 1. If you add a new file, submit an issue to the [production tracker](https://gitlab.com/gitlab-com/gl-infra/production/issues) or diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 0c7601b415e..4740cf4de7b 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -1,18 +1,16 @@ # Migration Style Guide When writing migrations for GitLab, you have to take into account that -these will be ran by hundreds of thousands of organizations of all sizes, some with +these will be run by hundreds of thousands of organizations of all sizes, some with many years of data in their database. In addition, having to take a server offline for an upgrade small or big is a -big burden for most organizations. For this reason it is important that your -migrations are written carefully, can be applied online and adhere to the style +big burden for most organizations. For this reason, it is important that your +migrations are written carefully, can be applied online, and adhere to the style guide below. Migrations are **not** allowed to require GitLab installations to be taken -offline unless _absolutely necessary_. Downtime assumptions should be based on -the behaviour of a migration when performed using PostgreSQL, as various -operations in MySQL may require downtime without there being alternatives. +offline unless _absolutely necessary_. When downtime is necessary the migration has to be approved by: @@ -87,7 +85,38 @@ be possible to downgrade in case of a vulnerability or bugs. In your migration, add a comment describing how the reversibility of the migration was tested. -## Multi Threading +## Atomicity + +By default, migrations are single transaction. That is, a transaction is opened +at the beginning of the migration, and committed after all steps are processed. + +Running migrations in a single transaction makes sure that if one of the steps fails, +none of the steps will be executed, leaving the database in valid state. +Therefore, either: + +- Put all migrations in one single-transaction migration. +- If necessary, put most actions in one migration and create a separate migration + for the steps that cannot be done in a single transaction. + +For example, if you create an empty table and need to build an index for it, +it is recommended to use a regular single-transaction migration and the default +rails schema statement: [`add_index`](https://api.rubyonrails.org/v5.2/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-add_index). +This is a blocking operation, but it won't cause problems because the table is not yet used, +and therefore it does not have any records yet. + +## Heavy operations in a single transaction + +When using a single-transaction migration, a transaction will hold on a database connection +for the duration of the migration, so you must make sure the actions in the migration +do not take too much time: In general, queries executed in a migration need to fit comfortably +within `15s` on GitLab.com. + +In case you need to insert, update, or delete a significant amount of data, you: + +- Must disable the single transaction with `disable_ddl_transaction!`. +- Should consider doing it in a [Background Migration](background_migrations.md). + +## Multi-Threading Sometimes a migration might need to use multiple Ruby threads to speed up a migration. For this to work your migration needs to include the module @@ -124,16 +153,16 @@ pool. This ensures each thread has its own connection object, and won't time out when trying to obtain one. **NOTE:** PostgreSQL has a maximum amount of connections that it allows. This -limit can vary from installation to installation. As a result it's recommended -you do not use more than 32 threads in a single migration. Usually 4-8 threads +limit can vary from installation to installation. As a result, it's recommended +you do not use more than 32 threads in a single migration. Usually, 4-8 threads should be more than enough. ## Removing indexes -When removing an index make sure to use the method `remove_concurrent_index` instead -of the regular `remove_index` method. The `remove_concurrent_index` method -automatically drops concurrent indexes when using PostgreSQL, removing the -need for downtime. To use this method you must disable single-transaction mode +If the table is not empty when removing an index, make sure to use the method +`remove_concurrent_index` instead of the regular `remove_index` method. +The `remove_concurrent_index` method drops indexes concurrently, so no locking is required, +and there is no need for downtime. To use this method, you must disable single-transaction mode by calling the method `disable_ddl_transaction!` in the body of your migration class like so: @@ -151,19 +180,25 @@ end Note that it is not necessary to check if the index exists prior to removing it. +For a small table (such as an empty one or one with less than `1,000` records), +it is recommended to use `remove_index` in a single-transaction migration, +combining it with other operations that don't require `disable_ddl_transaction!`. + ## Adding indexes -If you need to add a unique index please keep in mind there is the possibility +If you need to add a unique index, please keep in mind there is the possibility of existing duplicates being present in the database. This means that should always _first_ add a migration that removes any duplicates, before adding the unique index. -When adding an index make sure to use the method `add_concurrent_index` instead -of the regular `add_index` method. The `add_concurrent_index` method -automatically creates concurrent indexes when using PostgreSQL, removing the -need for downtime. To use this method you must disable transactions by calling -the method `disable_ddl_transaction!` in the body of your migration class like -so: +When adding an index to a non-empty table make sure to use the method +`add_concurrent_index` instead of the regular `add_index` method. +The `add_concurrent_index` method automatically creates concurrent indexes +when using PostgreSQL, removing the need for downtime. + +To use this method, you must disable single-transactions mode +by calling the method `disable_ddl_transaction!` in the body of your migration +class like so: ```ruby class MyMigration < ActiveRecord::Migration[4.2] @@ -181,16 +216,20 @@ class MyMigration < ActiveRecord::Migration[4.2] end ``` +For a small table (such as an empty one or one with less than `1,000` records), +it is recommended to use `add_index` in a single-transaction migration, combining it with other +operations that don't require `disable_ddl_transaction!`. + ## Adding foreign-key constraints -When adding a foreign-key constraint to either an existing or new -column remember to also add a index on the column. +When adding a foreign-key constraint to either an existing or a new column also +remember to add an index on the column. This is **required** for all foreign-keys, e.g., to support efficient cascading deleting: when a lot of rows in a table get deleted, the referenced records need to be deleted too. The database has to look for corresponding records in the referenced table. Without an index, this will result in a sequential scan on the -table which can take a long time. +table, which can take a long time. Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. @@ -204,13 +243,17 @@ class Migration < ActiveRecord::Migration[4.2] end ``` -When adding a foreign-key constraint to an existing column, we -have to employ `add_concurrent_foreign_key` and `add_concurrent_index` +When adding a foreign-key constraint to an existing column in a non-empty table, +we have to employ `add_concurrent_foreign_key` and `add_concurrent_index` instead of `add_reference`. +For an empty table (such as a fresh one), it is recommended to use +`add_reference` in a single-transaction migration, combining it with other +operations that don't require `disable_ddl_transaction!`. + ## Adding Columns With Default Values -When adding columns with default values you must use the method +When adding columns with default values to non-empty tables, you must use `add_column_with_default`. This method ensures the table is updated without requiring downtime. This method is not reversible so you must manually define the `up` and `down` methods in your migration class. @@ -234,10 +277,14 @@ end ``` Keep in mind that this operation can easily take 10-15 minutes to complete on -larger installations (e.g. GitLab.com). As a result you should only add default -values if absolutely necessary. There is a RuboCop cop that will fail if this -method is used on some tables that are very large on GitLab.com, which would -cause other issues. +larger installations (e.g. GitLab.com). As a result, you should only add +default values if absolutely necessary. There is a RuboCop cop that will fail if +this method is used on some tables that are very large on GitLab.com, which +would cause other issues. + +For a small table (such as an empty one or one with less than `1,000` records), +use `add_column` and `change_column_default` in a single-transaction migration, +combining it with other operations that don't require `disable_ddl_transaction!`. ## Updating an existing column @@ -255,8 +302,10 @@ update_column_in_batches(:projects, :foo, 10) do |table, query| end ``` -To perform a computed update, the value can be wrapped in `Arel.sql`, so Arel -treats it as an SQL literal. The below example is the same as the one above, but +If a computed update is needed, the value can be wrapped in `Arel.sql`, so Arel +treats it as an SQL literal. It's also a required deprecation for [Rails 6](https://gitlab.com/gitlab-org/gitlab-ce/issues/61451). + +The below example is the same as the one above, but the value is set to the product of the `bar` and `baz` columns: ```ruby @@ -277,12 +326,12 @@ staging environment - or asking someone else to do so for you - beforehand. By default, an integer column can hold up to a 4-byte (32-bit) number. That is a max value of 2,147,483,647. Be aware of this when creating a column that will -hold file sizes in byte units. If you are tracking file size in bytes this +hold file sizes in byte units. If you are tracking file size in bytes, this restricts the maximum file size to just over 2GB. To allow an integer column to hold up to an 8-byte (64-bit) number, explicitly set the limit to 8-bytes. This will allow the column to hold a value up to -9,223,372,036,854,775,807. +`9,223,372,036,854,775,807`. Rails migration example: @@ -296,9 +345,11 @@ add_column(:projects, :foo, :integer, default: 10, limit: 8) ## Timestamp column type -By default, Rails uses the `timestamp` data type that stores timestamp data without timezone information. -The `timestamp` data type is used by calling either the `add_timestamps` or the `timestamps` method. -Also Rails converts the `:datetime` data type to the `timestamp` one. +By default, Rails uses the `timestamp` data type that stores timestamp data +without timezone information. The `timestamp` data type is used by calling +either the `add_timestamps` or the `timestamps` method. + +Also, Rails converts the `:datetime` data type to the `timestamp` one. Example: @@ -319,14 +370,16 @@ def up end ``` -Instead of using these methods one should use the following methods to store timestamps with timezones: +Instead of using these methods, one should use the following methods to store +timestamps with timezones: - `add_timestamps_with_timezone` - `timestamps_with_timezone` -This ensures all timestamps have a time zone specified. This in turn means existing timestamps won't -suddenly use a different timezone when the system's timezone changes. It also makes it very clear which -timezone was used in the first place. +This ensures all timestamps have a time zone specified. This, in turn, means +existing timestamps won't suddenly use a different timezone when the system's +timezone changes. It also makes it very clear which timezone was used in the +first place. ## Storing JSON in database @@ -343,10 +396,7 @@ class AddOptionsToBuildMetadata < ActiveRecord::Migration[5.0] end ``` -On MySQL the `JSON` and `JSONB` is translated to `TEXT 1MB`, as `JSONB` is PostgreSQL only feature. - -For above reason you have to use a serializer to provide a translation layer -in order to support PostgreSQL and MySQL seamlessly: +You have to use a serializer to provide a translation layer: ```ruby class BuildMetadata @@ -356,7 +406,7 @@ end ## Testing -Make sure that your migration works with MySQL and PostgreSQL with data. An +Make sure that your migration works for databases with data. An empty database does not guarantee that your migration is correct. Make sure your migration can be reversed. @@ -364,7 +414,7 @@ Make sure your migration can be reversed. ## Data migration Please prefer Arel and plain SQL over usual ActiveRecord syntax. In case of -using plain SQL you need to quote all input manually with `quote_string` helper. +using plain SQL, you need to quote all input manually with `quote_string` helper. Example with Arel: @@ -389,7 +439,7 @@ select_all("SELECT name, COUNT(id) as cnt FROM tags GROUP BY name HAVING COUNT(i end ``` -If you need more complex logic you can define and use models local to a +If you need more complex logic, you can define and use models local to a migration. For example: ```ruby @@ -400,13 +450,13 @@ class MyMigration < ActiveRecord::Migration[4.2] end ``` -When doing so be sure to explicitly set the model's table name so it's not +When doing so be sure to explicitly set the model's table name, so it's not derived from the class name or namespace. ### Renaming reserved paths -When a new route for projects is introduced that could conflict with any -existing records. The path for this records should be renamed, and the +When a new route for projects is introduced, it could conflict with any +existing records. The path for these records should be renamed, and the related data should be moved on disk. Since we had to do this a few times already, there are now some helpers to help diff --git a/doc/development/module_with_instance_variables.md b/doc/development/module_with_instance_variables.md index 7bdfa04fc57..443eee0b62c 100644 --- a/doc/development/module_with_instance_variables.md +++ b/doc/development/module_with_instance_variables.md @@ -1,6 +1,6 @@ -## Modules with instance variables could be considered harmful +# Modules with instance variables could be considered harmful -### Background +## Background Rails somehow encourages people using modules and instance variables everywhere. For example, using instance variables in the controllers, @@ -9,7 +9,7 @@ helpers, and views. They're also encouraging the use of saving everything in a giant, single object, and people could access everything in that one giant object. -### The problems +## The problems Of course this is convenient to develop, because we just have everything within reach. However this has a number of downsides when that chosen object @@ -24,7 +24,7 @@ manipulated from 3 different modules. It's hard to track when those variables start giving us troubles. We don't know which module would suddenly change one of the variables. Everything could touch anything. -### Similar concerns +## Similar concerns People are saying multiple inheritance is bad. Mixing multiple modules with multiple instance variables scattering everywhere suffer from the same issue. @@ -40,7 +40,7 @@ Note that `included` doesn't solve the whole issue. They define the dependencies, but they still allow each modules to talk implicitly via the instance variables in the final giant object, and that's where the problem is. -### Solutions +## Solutions We should split the giant object into multiple objects, and they communicate with each other with the API, i.e. public methods. In short, composition over @@ -53,7 +53,7 @@ With clearly defined API, this would make things less coupled and much easier to debug and track, and much more extensible for other objects to use, because they communicate in a clear way, rather than implicit dependencies. -### Acceptable use +## Acceptable use However, it's not always bad to use instance variables in a module, as long as it's contained in the same module; that is, no other modules or @@ -74,7 +74,7 @@ Unfortunately it's not easy to code more complex rules into the cop, so we rely on people's best judgement. If we could find another good pattern we could easily add to the cop, we should do it. -### How to rewrite and avoid disabling this cop +## How to rewrite and avoid disabling this cop Even if we could just disable the cop, we should avoid doing so. Some code could be easily rewritten in simple form. Consider this acceptable method: @@ -181,7 +181,7 @@ rather than whatever includes the module, and those modules which were also included, making it much easier to track down any issues, and reducing the chance of having name conflicts. -### How to disable this cop +## How to disable this cop Put the disabling comment right after your code in the same line: @@ -210,14 +210,14 @@ end Note that you need to enable it at some point, otherwise everything below won't be checked. -### Things we might need to ignore right now +## Things we might need to ignore right now Because of the way Rails helpers and mailers work, we might not be able to avoid the use of instance variables there. For those cases, we could ignore them at the moment. At least we're not going to share those modules with other random objects, so they're still somewhat isolated. -### Instance variables in views +## Instance variables in views They're bad because we can't easily tell who's using the instance variables (from controller's point of view) and where we set them up (from partials' diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md new file mode 100644 index 00000000000..2c7e5935435 --- /dev/null +++ b/doc/development/namespaces_storage_statistics.md @@ -0,0 +1,178 @@ +# Database case study: Namespaces storage statistics + +## Introduction + +On [Storage and limits management for groups](https://gitlab.com/groups/gitlab-org/-/epics/886), +we want to facilitate a method for easily viewing the amount of +storage consumed by a group, and allow easy management. + +## Proposal + +1. Create a new ActiveRecord model to hold the namespaces' statistics in an aggregated form (only for root namespaces). +1. Refresh the statistics in this model every time a project belonging to this namespace is changed. + +## Problem + +In GitLab, we update the project storage statistics through a +[callback](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/project.rb#L90) +every time the project is saved. + +The summary of those statistics per namespace is then retrieved +by [`Namespaces#with_statistics`](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/namespace.rb#L70) scope. Analyzing this query we noticed that: + +- It takes up to `1.2` seconds for namespaces with over `15k` projects. +- It can't be analyzed with [ChatOps](chatops_on_gitlabcom.md), as it times out. + +Additionally, the pattern that is currently used to update the project statistics +(the callback) doesn't scale adequately. It is currently one of the largest +[database queries transactions on production](https://gitlab.com/gitlab-org/gitlab-ce/issues/62488) +that takes the most time overall. We can't add one more query to it as +it will increase the transaction's length. + +Because of all of the above, we can't apply the same pattern to store +and update the namespaces statistics, as the `namespaces` table is one +of the largest tables on GitLab.com. Therefore we needed to find a performant and +alternative method. + +## Attempts + +### Attempt A: PostgreSQL materialized view + +Model can be updated through a refresh strategy based on a project routes SQL and a [materialized view](https://www.postgresql.org/docs/9.6/rules-materializedviews.html): + +```sql +SELECT split_part("rs".path, '/', 1) as root_path, + COALESCE(SUM(ps.storage_size), 0) AS storage_size, + COALESCE(SUM(ps.repository_size), 0) AS repository_size, + COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, + COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, + COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size +FROM "projects" + INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' + INNER JOIN project_statistics ps ON ps.project_id = projects.id +GROUP BY root_path +``` + +We could then execute the query with: + +```sql +REFRESH MATERIALIZED VIEW root_namespace_storage_statistics; +``` + +While this implied a single query update (and probably a fast one), it has some downsides: + +- Materialized views syntax varies from PostgreSQL and MySQL. While this feature was worked on, MySQL was still supported by GitLab. +- Rails does not have native support for materialized views. We'd need to use a specialized gem to take care of the management of the database views, which implies additional work. + +### Attempt B: An update through a CTE + +Similar to Attempt A: Model update done through a refresh strategy with a [Common Table Expression](https://www.postgresql.org/docs/9.1/queries-with.html) + +```sql +WITH refresh AS ( + SELECT split_part("rs".path, '/', 1) as root_path, + COALESCE(SUM(ps.storage_size), 0) AS storage_size, + COALESCE(SUM(ps.repository_size), 0) AS repository_size, + COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, + COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, + COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size + FROM "projects" + INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' + INNER JOIN project_statistics ps ON ps.project_id = projects.id + GROUP BY root_path) +UPDATE namespace_storage_statistics +SET storage_size = refresh.storage_size, + repository_size = refresh.repository_size, + wiki_size = refresh.wiki_size, + lfs_objects_size = refresh.lfs_objects_size, + build_artifacts_size = refresh.build_artifacts_size, + packages_size = refresh.packages_size +FROM refresh + INNER JOIN routes rs ON rs.path = refresh.root_path AND rs.source_type = 'Namespace' +WHERE namespace_storage_statistics.namespace_id = rs.source_id +``` + +Same benefits and downsides as attempt A. + +### Attempt C: Get rid of the model and store the statistics on Redis + +We could get rid of the model that stores the statistics in aggregated form and instead use a Redis Set. +This would be the [boring solution](https://about.gitlab.com/handbook/values/#boring-solutions) and the fastest one +to implement, as GitLab already includes Redis as part of its [Architecture](architecture.md#redis). + +The downside of this approach is that Redis does not provide the same persistence/consistency guarantees as PostgreSQL, +and this is information we can't afford to lose in a Redis failure. + +### Attempt D: Tag the root namespace and its child namespaces + +Directly relate the root namespace to its child namespaces, so +whenever a namespace is created without a parent, this one is tagged +with the root namespace ID: + +| id | root_id | parent_id +|:---|:--------|:---------- +| 1 | 1 | NULL +| 2 | 1 | 1 +| 3 | 1 | 2 + +To aggregate the statistics inside a namespace we'd execute something like: + +```sql +SELECT COUNT(...) +FROM projects +WHERE namespace_id IN ( + SELECT id + FROM namespaces + WHERE root_id = X +) +``` + +Even though this approach would make aggregating much easier, it has some major downsides: + +- We'd have to migrate **all namespaces** by adding and filling a new column. Because of the size of the table, dealing with time/cost will not be great. The background migration will take approximately `153h`, see <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29772>. +- Background migration has to be shipped one release before, delaying the functionality by another milestone. + +### Attempt E (final): Update the namespace storage statistics in async way + +This approach consists of keep using the incremental statistics updates we currently already have, +but we refresh them through Sidekiq jobs and in different transactions: + +1. Create a second table (`namespace_aggregation_schedules`) with two columns `id` and `namespace_id`. +1. Whenever the statistics of a project changes, insert a row into `namespace_aggregation_schedules` + - We don't insert a new row if there's already one related to the root namespace. + - Keeping in mind the length of the transaction that involves updating `project_statistics`(<https://gitlab.com/gitlab-org/gitlab-ce/issues/62488>), the insertion should be done in a different transaction and through a Sidekiq Job. +1. After inserting the row, we schedule another worker to be executed async at two different moments: + - One enqueued for immediate execution and another one scheduled in `1.5h` hours. + - We only schedule the jobs, if we can obtain a `1.5h` lease on Redis on a key based on the root namespace ID. + - If we can't obtain the lease, it indicates there's another aggregation already in progress, or scheduled in no more than `1.5h`. +1. This worker will: + - Update the root namespace storage statistics by querying all the namespaces through a service. + - Delete the related `namespace_aggregation_schedules` after the update. +1. Another Sidekiq job is also included to traverse any remaining rows on the `namespace_aggregation_schedules` table and schedule jobs for every pending row. + - This job is scheduled with cron to run every night (UTC). + +This implementation has the following benefits: + +- All the updates are done async, so we're not increasing the length of the transactions for `project_statistics`. +- We're doing the update in a single SQL query. +- It is compatible with PostgreSQL and MySQL. +- No background migration required. + +The only downside of this approach is that namespaces' statistics are updated up to `1.5` hours after the change is done, +which means there's a time window in which the statistics are inaccurate. Because we're still not +[enforcing storage limits](https://gitlab.com/gitlab-org/gitlab-ce/issues/30421), this is not a major problem. + +## Conclusion + +Updating the storage statistics asynchronously, was the less problematic and +performant approach of aggregating the root namespaces. + +All the details regarding this use case can be found on: + +- <https://gitlab.com/gitlab-org/gitlab-ce/issues/62214> +- Merge Request with the implementation: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28996> + +Performance of the namespace storage statistics were measured in staging and production (GitLab.com). All results were posted +on <https://gitlab.com/gitlab-org/gitlab-ce/issues/64092>: No problem has been reported so far. diff --git a/doc/development/new_fe_guide/dependencies.md b/doc/development/new_fe_guide/dependencies.md index 12a4f089d41..8a6930acd37 100644 --- a/doc/development/new_fe_guide/dependencies.md +++ b/doc/development/new_fe_guide/dependencies.md @@ -15,6 +15,18 @@ Exceptions are made for some tools that we require in the `gitlab:assets:compile` CI job such as `webpack-bundle-analyzer` to analyze our production assets post-compile. +To add or upgrade a dependency, run: + +```sh +yarn add <your dependency here> +``` + +This may introduce duplicate dependencies. To de-duplicate `yarn.lock`, run: + +```sh +node_modules/.bin/yarn-deduplicate --list --strategy fewer yarn.lock && yarn install +``` + --- > TODO: Add Dependencies diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md index 81a29170129..ae5c4c6a6cc 100644 --- a/doc/development/new_fe_guide/development/accessibility.md +++ b/doc/development/new_fe_guide/development/accessibility.md @@ -1,17 +1,21 @@ # Accessiblity + Using semantic HTML plays a key role when it comes to accessibility. ## Accessible Rich Internet Applications - ARIA + WAI-ARIA, the Accessible Rich Internet Applications specification, defines a way to make Web content and Web applications more accessible to people with disabilities. > Note: It is [recommended][using-aria] to use semantic elements as the primary method to achieve accessibility rather than adding aria attributes. Adding aria attributes should be seen as a secondary method for creating accessible elements. ### Role + The `role` attribute describes the role the element plays in the context of the document. Check the list of WAI-ARIA roles [here][roles] ## Icons + When using icons or images that aren't absolutely needed to understand the context, we should use `aria-hidden="true"`. On the other hand, if an icon is crucial to understand the context we should do one of the following: @@ -20,6 +24,7 @@ On the other hand, if an icon is crucial to understand the context we should do 1. Use `aria-labelledby` to point to an element that contains the explanation for that icon ## Form inputs + In forms we should use the `for` attribute in the label statement: ``` diff --git a/doc/development/new_fe_guide/development/performance.md b/doc/development/new_fe_guide/development/performance.md index c54b8305991..d41239693bf 100644 --- a/doc/development/new_fe_guide/development/performance.md +++ b/doc/development/new_fe_guide/development/performance.md @@ -5,7 +5,7 @@ We have a performance dashboard available in one of our [grafana instances](https://dashboards.gitlab.net/d/1EBTz3Dmz/sitespeed-page-summary?orgId=1). This dashboard automatically aggregates metric data from [sitespeed.io](https://www.sitespeed.io/) every 6 hours. These changes are displayed after a set number of pages are aggregated. These pages can be found inside a text file in the gitlab-build-images [repository](https://gitlab.com/gitlab-org/gitlab-build-images) called [gitlab.txt](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/scripts/gitlab.txt) -Any frontend engineer can contribute to this dashboard. They can contribute by adding or removing urls of pages from this text file. Please have a [frontend monitoring expert](https://about.gitlab.com/company/team) review your changes before assigning to a maintainer of the `gitlab-build-images` project. The changes will go live on the next scheduled run after the changes are merged into `master`. +Any frontend engineer can contribute to this dashboard. They can contribute by adding or removing urls of pages from this text file. Please have a [frontend monitoring expert](https://about.gitlab.com/company/team/) review your changes before assigning to a maintainer of the `gitlab-build-images` project. The changes will go live on the next scheduled run after the changes are merged into `master`. There are 3 recommended high impact metrics to review on each page: diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md index 2b62c2a41fe..e0d413b748b 100644 --- a/doc/development/new_fe_guide/development/testing.md +++ b/doc/development/new_fe_guide/development/testing.md @@ -1,361 +1,6 @@ -# Overview of Frontend Testing +--- +redirect_to: '../../testing_guide/frontend_testing.md' +--- -Tests relevant for frontend development can be found at the following places: +This document was moved to [another location](../../testing_guide/frontend_testing.md). -- `spec/javascripts/` which are run by Karma (command: `yarn karma`) and contain - - [frontend unit tests](#frontend-unit-tests) - - [frontend component tests](#frontend-component-tests) - - [frontend integration tests](#frontend-integration-tests) -- `spec/frontend/` which are run by Jest (command: `yarn jest`) and contain - - [frontend unit tests](#frontend-unit-tests) - - [frontend component tests](#frontend-component-tests) - - [frontend integration tests](#frontend-integration-tests) -- `spec/features/` which are run by RSpec and contain - - [feature tests](#feature-tests) - -All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-ce/issues/52483)). - -In addition there were feature tests in `features/` run by Spinach in the past. -These have been removed from our codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-ce/issues/23036)). - -See also: - -- [Old testing guide](../../testing_guide/frontend_testing.html). -- [Notes on testing Vue components](../../fe_guide/vue.html#testing-vue-components). - -## Frontend unit tests - -Unit tests are on the lowest abstraction level and typically test functionality that is not directly perceivable by a user. - -### When to use unit tests - -<details> - <summary>exported functions and classes</summary> - Anything that is exported can be reused at various places in a way you have no control over. - Therefore it is necessary to document the expected behavior of the public interface with tests. -</details> - -<details> - <summary>Vuex actions</summary> - Any Vuex action needs to work in a consistent way independent of the component it is triggered from. -</details> - -<details> - <summary>Vuex mutations</summary> - For complex Vuex mutations it helps to identify the source of a problem by separating the tests from other parts of the Vuex store. -</details> - -### When *not* to use unit tests - -<details> - <summary>non-exported functions or classes</summary> - Anything that is not exported from a module can be considered private or an implementation detail and doesn't need to be tested. -</details> - -<details> - <summary>constants</summary> - Testing the value of a constant would mean to copy it. - This results in extra effort without additional confidence that the value is correct. -</details> - -<details> - <summary>Vue components</summary> - Computed properties, methods, and lifecycle hooks can be considered an implementation detail of components and don't need to be tested. - They are implicitly covered by component tests. - The <a href="https://vue-test-utils.vuejs.org/guides/#getting-started">official Vue guidelines</a> suggest the same. -</details> - -### What to mock in unit tests - -<details> - <summary>state of the class under test</summary> - Modifying the state of the class under test directly rather than using methods of the class avoids side-effects in test setup. -</details> - -<details> - <summary>other exported classes</summary> - Every class needs to be tested in isolation to prevent test scenarios from growing exponentially. -</details> - -<details> - <summary>single DOM elements if passed as parameters</summary> - For tests that only operate on single DOM elements rather than a whole page, creating these elements is cheaper than loading a whole HTML fixture. -</details> - -<details> - <summary>all server requests</summary> - When running frontend unit tests, the backend may not be reachable. - Therefore all outgoing requests need to be mocked. -</details> - -<details> - <summary>asynchronous background operations</summary> - Background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. -</details> - -### What *not* to mock in unit tests - -<details> - <summary>non-exported functions or classes</summary> - Everything that is not exported can be considered private to the module and will be implicitly tested via the exported classes / functions. -</details> - -<details> - <summary>methods of the class under test</summary> - By mocking methods of the class under test, the mocks will be tested and not the real methods. -</details> - -<details> - <summary>utility functions (pure functions, or those that only modify parameters)</summary> - If a function has no side effects because it has no state, it is safe to not mock it in tests. -</details> - -<details> - <summary>full HTML pages</summary> - Loading the HTML of a full page slows down tests, so it should be avoided in unit tests. -</details> - -## Frontend component tests - -Component tests cover the state of a single component that is perceivable by a user depending on external signals such as user input, events fired from other components, or application state. - -### When to use component tests - -- Vue components - -### When *not* to use component tests - -<details> - <summary>Vue applications</summary> - Vue applications may contain many components. - Testing them on a component level requires too much effort. - Therefore they are tested on frontend integration level. -</details> - -<details> - <summary>HAML templates</summary> - HAML templates contain only Markup and no frontend-side logic. - Therefore they are not complete components. -</details> - -### What to mock in component tests - -<details> - <summary>DOM</summary> - Operating on the real DOM is significantly slower than on the virtual DOM. -</details> - -<details> - <summary>properties and state of the component under test</summary> - Similarly to testing classes, modifying the properties directly (rather than relying on methods of the component) avoids side-effects. -</details> - -<details> - <summary>Vuex store</summary> - To avoid side effects and keep component tests simple, Vuex stores are replaced with mocks. -</details> - -<details> - <summary>all server requests</summary> - Similar to unit tests, when running component tests, the backend may not be reachable. - Therefore all outgoing requests need to be mocked. -</details> - -<details> - <summary>asynchronous background operations</summary> - Similar to unit tests, background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. -</details> - -<details> - <summary>child components</summary> - Every component is tested individually, so child components are mocked. - See also <a href="https://vue-test-utils.vuejs.org/api/#shallowmount">shallowMount()</a> -</details> - -### What *not* to mock in component tests - -<details> - <summary>methods or computed properties of the component under test</summary> - By mocking part of the component under test, the mocks will be tested and not the real component. -</details> - -<details> - <summary>functions and classes independent from Vue</summary> - All plain JavaScript code is already covered by unit tests and needs not to be mocked in component tests. -</details> - -## Frontend integration tests - -Integration tests cover the interaction between all components on a single page. -Their abstraction level is comparable to how a user would interact with the UI. - -### When to use integration tests - -<details> - <summary>page bundles (<code>index.js</code> files in <code>app/assets/javascripts/pages/</code>)</summary> - Testing the page bundles ensures the corresponding frontend components integrate well. -</details> - -<details> - <summary>Vue applications outside of page bundles</summary> - Testing Vue applications as a whole ensures the corresponding frontend components integrate well. -</details> - -### What to mock in integration tests - -<details> - <summary>HAML views (use fixtures instead)</summary> - Rendering HAML views requires a Rails environment including a running database which we cannot rely on in frontend tests. -</details> - -<details> - <summary>all server requests</summary> - Similar to unit and component tests, when running component tests, the backend may not be reachable. - Therefore all outgoing requests need to be mocked. -</details> - -<details> - <summary>asynchronous background operations that are not perceivable on the page</summary> - Background operations that affect the page need to be tested on this level. - All other background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. -</details> - -### What *not* to mock in integration tests - -<details> - <summary>DOM</summary> - Testing on the real DOM ensures our components work in the environment they are meant for. - Part of this will be delegated to <a href="https://gitlab.com/gitlab-org/quality/team-tasks/issues/45">cross-browser testing</a>. -</details> - -<details> - <summary>properties or state of components</summary> - On this level, all tests can only perform actions a user would do. - For example to change the state of a component, a click event would be fired. -</details> - -<details> - <summary>Vuex stores</summary> - When testing the frontend code of a page as a whole, the interaction between Vue components and Vuex stores is covered as well. -</details> - -## Feature tests - -In contrast to [frontend integration tests](#frontend-integration-tests), feature tests make requests against the real backend instead of using fixtures. -This also implies that database queries are executed which makes this category significantly slower. - -See also the [RSpec testing guidelines](../../testing_guide/best_practices.md#rspec). - -### When to use feature tests - -- use cases that require a backend and cannot be tested using fixtures -- behavior that is not part of a page bundle but defined globally - -### Relevant notes - -A `:js` flag is added to the test to make sure the full environment is loaded. - -``` -scenario 'successfully', :js do - sign_in(create(:admin)) -end -``` - -The steps of each test are written using capybara methods ([documentation](https://www.rubydoc.info/gems/capybara/2.15.1)). - -Bear in mind <abbr title="XMLHttpRequest">XHR</abbr> calls might require you to use `wait_for_requests` in between steps, like so: - -```rspec -find('.form-control').native.send_keys(:enter) - -wait_for_requests - -expect(page).not_to have_selector('.card') -``` - -## Test helpers - -### Vuex Helper: `testAction` - -We have a helper available to make testing actions easier, as per [official documentation](https://vuex.vuejs.org/guide/testing.html): - -``` -testAction( - actions.actionName, // action - { }, // params to be passed to action - state, // state - [ - { type: types.MUTATION}, - { type: types.MUTATION_1, payload: {}}, - ], // mutations committed - [ - { type: 'actionName', payload: {}}, - { type: 'actionName1', payload: {}}, - ] // actions dispatched - done, -); -``` - -Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/javascripts/ide/stores/actions_spec.js). - -### Vue Helper: `mountComponent` - -To make mounting a Vue component easier and more readable, we have a few helpers available in `spec/helpers/vue_mount_component_helper`. - -- `createComponentWithStore` -- `mountComponentWithStore` - -Examples of usage: - -``` -beforeEach(() => { - vm = createComponentWithStore(Component, store); - - vm.$store.state.currentBranchId = 'master'; - - vm.$mount(); -}, -``` - -``` -beforeEach(() => { - vm = mountComponentWithStore(Component, { - el: '#dummy-element', - store, - props: { badge }, - }); -}, -``` - -Don't forget to clean up: - -``` -afterEach(() => { - vm.$destroy(); -}); -``` - -## Testing with older browsers - -Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or Browserstack using the following steps: - -### Browserstack - -[Browserstack](https://www.browserstack.com/) allows you to test more than 1200 mobile devices and browsers. -You can use it directly through the [live app](https://www.browserstack.com/live) or you can install the [chrome extension](https://chrome.google.com/webstore/detail/browserstack/nkihdmlheodkdfojglpcjjmioefjahjb) for easy access. -You can find the credentials on 1Password, under `frontendteam@gitlab.com`. - -### Firefox - -#### macOS - -You can download any older version of Firefox from the releases FTP server, <https://ftp.mozilla.org/pub/firefox/releases/> - -1. From the website, select a version, in this case `50.0.1`. -1. Go to the mac folder. -1. Select your preferred language, you will find the dmg package inside, download it. -1. Drag and drop the application to any other folder but the `Applications` folder. -1. Rename the application to something like `Firefox_Old`. -1. Move the application to the `Applications` folder. -1. Open up a terminal and run `/Applications/Firefox_Old.app/Contents/MacOS/firefox-bin -profilemanager` to create a new profile specific to that Firefox version. -1. Once the profile has been created, quit the app, and run it again like normal. You now have a working older Firefox version. diff --git a/doc/development/new_fe_guide/index.md b/doc/development/new_fe_guide/index.md index 0e8f5486861..227d03bd86f 100644 --- a/doc/development/new_fe_guide/index.md +++ b/doc/development/new_fe_guide/index.md @@ -19,7 +19,6 @@ Learn about all the internal JavaScript modules that make up our frontend. Style guides to keep our code consistent. - ## [Tips](tips.md) Tips from our frontend team to develop more efficiently and effectively. diff --git a/doc/development/new_fe_guide/modules/dirty_submit.md b/doc/development/new_fe_guide/modules/dirty_submit.md index 6c03958b463..217743ea395 100644 --- a/doc/development/new_fe_guide/modules/dirty_submit.md +++ b/doc/development/new_fe_guide/modules/dirty_submit.md @@ -1,7 +1,6 @@ # Dirty Submit -> [Introduced][ce-21115] in GitLab 11.3. -> [dirty_submit][dirty-submit] +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21115) in GitLab 11.3. ## Summary @@ -9,6 +8,9 @@ Prevent submitting forms with no changes. Currently handles `input`, `textarea` and `select` elements. +Also, see [the code](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/dirty_submit/) +within the GitLab project. + ## Usage ```js @@ -18,6 +20,3 @@ new DirtySubmitForm(document.querySelector('form')); // or new DirtySubmitForm(document.querySelectorAll('form')); ``` - -[ce-21115]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21115 -[dirty-submit]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/dirty_submit/
\ No newline at end of file diff --git a/doc/development/new_fe_guide/style/index.md b/doc/development/new_fe_guide/style/index.md index 335d9e66240..f073dc56f1f 100644 --- a/doc/development/new_fe_guide/style/index.md +++ b/doc/development/new_fe_guide/style/index.md @@ -8,7 +8,7 @@ ## [Vue style guide](vue.md) -# Tooling +## Tooling ## [Prettier](prettier.md) diff --git a/doc/development/new_fe_guide/style/javascript.md b/doc/development/new_fe_guide/style/javascript.md index 3019eaa089c..b742d567f41 100644 --- a/doc/development/new_fe_guide/style/javascript.md +++ b/doc/development/new_fe_guide/style/javascript.md @@ -71,7 +71,6 @@ class myClass { } const instance = new myClass(); instance.makeRequest(); - ``` ## Avoid classes to handle DOM events @@ -189,8 +188,8 @@ disabled due to legacy compatibility reasons but they are in the process of bein Do not disable specific ESLint rules. Due to technical debt, you may disable the following rules only if you are invoking/instantiating existing code modules. - - [no-new](http://eslint.org/docs/rules/no-new) - - [class-method-use-this](http://eslint.org/docs/rules/class-methods-use-this) +- [no-new](http://eslint.org/docs/rules/no-new) +- [class-method-use-this](http://eslint.org/docs/rules/class-methods-use-this) > Note: Disable these rules on a per line basis. This makes it easier to refactor - in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`. +> in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`. diff --git a/doc/development/new_fe_guide/style/prettier.md b/doc/development/new_fe_guide/style/prettier.md index 4495f38f262..5f44c640d76 100644 --- a/doc/development/new_fe_guide/style/prettier.md +++ b/doc/development/new_fe_guide/style/prettier.md @@ -4,7 +4,7 @@ Our code is automatically formatted with [Prettier](https://prettier.io) to foll ## Editor -The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. +The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. Please take care that you only let Prettier format the same file types as the global Yarn script does (.js, .vue, and .scss). In VSCode by example you can easily exclude file formats in your settings file: @@ -28,6 +28,7 @@ Updates all currently staged files (based on `git diff`) with Prettier and saves ``` yarn prettier-staged ``` + Checks all currently staged files (based on `git diff`) with Prettier and log which files would need manual updating to the console. ``` diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md index 0ba354d28a2..ea5c18f1a8c 100644 --- a/doc/development/omnibus.md +++ b/doc/development/omnibus.md @@ -1,32 +1,32 @@ -# What you should know about omnibus packages +# What you should know about Omnibus packages -Most users install GitLab using our omnibus packages. As a developer it can be -good to know how the omnibus packages differ from what you have on your laptop +Most users install GitLab using our Omnibus packages. As a developer it can be +good to know how the Omnibus packages differ from what you have on your laptop when you are coding. ## Files are owned by root by default -All the files in the Rails tree (`app/`, `config/` etc.) are owned by 'root' in -omnibus installations. This makes the installation simpler and it provides -extra security. The omnibus reconfigure script contains commands that give -write access to the 'git' user only where needed. +All the files in the Rails tree (`app/`, `config/` etc.) are owned by `root` in +Omnibus installations. This makes the installation simpler and it provides +extra security. The Omnibus reconfigure script contains commands that give +write access to the `git` user only where needed. -For example, the 'git' user is allowed to write in the `log/` directory, in +For example, the `git` user is allowed to write in the `log/` directory, in `public/uploads`, and they are allowed to rewrite the `db/schema.rb` file. In other cases, the reconfigure script tricks GitLab into not trying to write a file. For instance, GitLab will generate a `.secret` file if it cannot find one -and write it to the Rails root. In the omnibus packages, reconfigure writes the +and write it to the Rails root. In the Omnibus packages, reconfigure writes the `.secret` file first, so that GitLab never tries to write it. ## Code, data and logs are in separate directories -The omnibus design separates code (read-only, under `/opt/gitlab`) from data +The Omnibus design separates code (read-only, under `/opt/gitlab`) from data (read/write, under `/var/opt/gitlab`) and logs (read/write, under `/var/log/gitlab`). To make this happen the reconfigure script sets custom paths where it can in GitLab config files, and where there are no path settings, it uses symlinks. For example, `config/gitlab.yml` is treated as data so that file is a symlink. -The same goes for `public/uploads`. The `log/` directory is replaced by omnibus +The same goes for `public/uploads`. The `log/` directory is replaced by Omnibus with a symlink to `/var/log/gitlab/gitlab-rails`. diff --git a/doc/development/performance.md b/doc/development/performance.md index c034f4a344b..14b3f8204d2 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -246,6 +246,7 @@ irb(main):002:0> results.last.attributes.keys irb(main):003:0> results.where(status: "passed").average(:time).to_s => "0.211340155844156" ``` + These results can also be placed into a PostgreSQL database by setting the `RSPEC_PROFILING_POSTGRES_URL` variable. This is used to profile the test suite when running in the CI environment. @@ -266,7 +267,7 @@ piece of code is worth optimizing. The only two things you can do are: 1. Think about what the code does, how it's used, how many times it's called and how much time is spent in it relative to the total execution time (e.g., the total time spent in a web request). -2. Ask others (preferably in the form of an issue). +1. Ask others (preferably in the form of an issue). Some examples of changes that aren't really important/worth the effort: @@ -284,10 +285,10 @@ directly in a web request as much as possible. This has numerous benefits such as: 1. An error won't prevent the request from completing. -2. The process being slow won't affect the loading time of a page. -3. In case of a failure it's easy to re-try the process (Sidekiq takes care of +1. The process being slow won't affect the loading time of a page. +1. In case of a failure it's easy to re-try the process (Sidekiq takes care of this automatically). -4. By isolating the code from a web request it will hopefully be easier to test +1. By isolating the code from a web request it will hopefully be easier to test and maintain. It's especially important to use Sidekiq as much as possible when dealing with diff --git a/doc/development/policies.md b/doc/development/policies.md index c4ac42bb40a..833b0acb13e 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -89,8 +89,8 @@ Each line represents a rule that was evaluated. There are a few things to note: 1. The `-` or `+` symbol indicates whether the rule block was evaluated to be `false` or `true`, respectively. -2. The number inside the brackets indicates the score. -3. The last part of the line (e.g. `@john : Issue/1`) shows the username +1. The number inside the brackets indicates the score. +1. The last part of the line (e.g. `@john : Issue/1`) shows the username and subject for that rule. Here you can see that the first four rules were evaluated `false` for diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 0511e735843..576601372a3 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -33,12 +33,10 @@ For example: you might be interested in migrating all dependent data to a differ class ImportCommonMetrics < ActiveRecord::Migration[4.2] include Gitlab::Database::MigrationHelpers - require Rails.root.join('db/importers/common_metrics_importer.rb') - DOWNTIME = false def up - Importers::CommonMetricsImporter.new.execute + ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute end def down diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index a6b60149ea4..3787e2ef187 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -36,6 +36,13 @@ it "avoids N+1 database queries" do end ``` +## Use request specs instead of controller specs + +Use a [request spec](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/spec/requests) when writing a N+1 test on the controller level. + +Controller specs should not be used to write N+1 tests as the controller is only initialized once per example. +This could lead to false successes where subsequent "requests" could have queries reduced (e.g. because of memoization). + ## Finding the source of the query It may be useful to identify the source of the queries by looking at the call backtrace. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index c97e179910b..e9d6cfe00b2 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -9,7 +9,7 @@ bundle exec rake setup ``` The `setup` task is an alias for `gitlab:setup`. -This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database. +This tasks calls `db:reset` to create the database, and calls `db:seed_fu` to seed the database. Note: `db:setup` calls `db:seed` but this does nothing. ### Seeding issues for all or a given project diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index f8c33ff2b85..dc51bf80e92 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -8,4 +8,4 @@ In December 2018, Tiago Botelho hosted a [Deep Dive] on GitLab's [Pull Repositor [Pull Repository Mirroring functionality]: ../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter [recording on YouTube]: https://www.youtube.com/watch?v=sSZq0fpdY-Y [Google Slides]: https://docs.google.com/presentation/d/17BTT6M6RyNRckV4wTt-dr07nIfBvD325_xVBoLtSoPM/edit?usp=sharing -[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf
\ No newline at end of file +[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 59da02ed6fd..fce144f8dc2 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -14,13 +14,13 @@ on maintainability, the ability to easily debug problems, or even performance. An example would be to use `ProjectsFinder` in `IssuesFinder` to limit issues to those belonging to a set of projects. While initially this may seem like a good idea, both classes provide a very high level interface with very little control. -This means that `IssuesFinder` may not be able to produce a better optimised +This means that `IssuesFinder` may not be able to produce a better optimized database query, as a large portion of the query is controlled by the internals of `ProjectsFinder`. To work around this problem, you would use the same code used by `ProjectsFinder`, instead of using `ProjectsFinder` itself directly. This allows -you to compose your behaviour better, giving you more control over the behaviour +you to compose your behavior better, giving you more control over the behavior of the code. To illustrate, consider the following code from `IssuableFinder#projects`: @@ -52,7 +52,7 @@ functionality is added to this (high level) interface. Instead of _only_ affecting the cases where this is necessary, it may also start affecting `IssuableFinder` in a negative way. For example, the query produced by `GroupProjectsFinder` may include unnecessary conditions. Since we're using a -finder here, we can't easily opt-out of that behaviour. We could add options to +finder here, we can't easily opt-out of that behavior. We could add options to do so, but then we'd need as many options as we have features. Every option adds two code paths, which means that for four features we have to cover 8 different code paths. @@ -213,6 +213,5 @@ The API provided by Active Record itself, such as the `where` method, `save`, Everything in `app/workers`. -The scheduling of Sidekiq jobs using `SomeWorker.perform_async`, `perform_in`, -etc. Directly invoking a worker using `SomeWorker.new.perform` should be avoided -at all times in application code, though this is fine to use in tests. +Use `SomeWorker.perform_async` or `SomeWorker.perform_in` to schedule Sidekiq +jobs. Never directly invoke a worker using `SomeWorker.new.perform`. diff --git a/doc/development/session.md b/doc/development/session.md index 9edce3dbda0..971795d8816 100644 --- a/doc/development/session.md +++ b/doc/development/session.md @@ -17,7 +17,7 @@ When storing values in a session it is best to: - Use simple primitives and avoid storing objects to avoid marshaling complications. - Clean up after unneeded variables to keep memory usage in Redis down. -## Gitlab::Session +## GitLab::Session Sometimes you might want to persist data in the session instead of another store like the database. `Gitlab::Session` lets you access this without passing the session around extensively. For example, you could access it from within a policy without having to pass the session through to each place permissions are checked from. diff --git a/doc/development/sha1_as_binary.md b/doc/development/sha1_as_binary.md index 3151cc29bbc..6c4252ec634 100644 --- a/doc/development/sha1_as_binary.md +++ b/doc/development/sha1_as_binary.md @@ -2,7 +2,7 @@ Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string requires at least 40 bytes, an additional byte to store the encoding, and -perhaps more space depending on the internals of PostgreSQL and MySQL. +perhaps more space depending on the internals of PostgreSQL. On the other hand, if one were to store a SHA1 as binary one would only need 20 bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index 7bdf676be58..1300c99622e 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -35,7 +35,7 @@ Gitlab::Popen.popen(%W(find /some/path -not -path /some/path -mmin +120 -delete) This coding style could have prevented CVE-2013-4490. -## Always use the configurable git binary path for git commands +## Always use the configurable Git binary path for Git commands ```ruby # Wrong @@ -114,7 +114,7 @@ user = `whoami` user, exit_status = Gitlab::Popen.popen(%W(whoami)) ``` -In other repositories, such as gitlab-shell you can also use `IO.popen`. +In other repositories, such as GitLab Shell you can also use `IO.popen`. ```ruby # Safe IO.popen example diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md new file mode 100644 index 00000000000..0809f8b1a0a --- /dev/null +++ b/doc/development/shell_scripting_guide/index.md @@ -0,0 +1,118 @@ +# Shell scripting standards and style guidelines + +## Overview + +GitLab consists of many various services and sub-projects. The majority of +their backend code is written in [Ruby](https://www.ruby-lang.org) and +[Go](https://golang.org). However, some of them use shell scripts for +automation of routine system administration tasks like deployment, +installation, etc. It's being done either for historical reasons or as an effort +to minimize the dependencies, for instance, for Docker images. + +This page aims to define and organize our shell scripting guidelines, +based on our various experiences. All shell scripts across GitLab project +should be eventually harmonized with this guide. If there are any per-project +deviations from this guide, they should be described in the +`README.md` or `PROCESS.md` file for such a project. + +### Avoid using shell scripts + +CAUTION: **Caution:** +This is a must-read section. + +Having said all of the above, we recommend staying away from shell scripts +as much as possible. A language like Ruby or Python (if required for +consistency with codebases that we leverage) is almost always a better choice. +The high-level interpreted languages have more readable syntax, offer much more +mature capabilities for unit-testing, linting, and error reporting. + +Use shell scripts only if there's a strong restriction on project's +dependencies size or any other requirements that are more important +in a particular case. + +## Scope of this guide + +According to the [GitLab installation requirements](../../install/requirements.md), +this guide covers only those shells that are used by +[supported Linux distributions](../../install/requirements.md#supported-linux-distributions), +that is: + +- [POSIX Shell](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html) +- [Bash](https://www.gnu.org/software/bash/) + +## Shell language choice + +- When you need to reduce the dependencies list, use what's provided by the environment. For example, for Docker images it's `sh` from `alpine` which is the base image for most of our tool images. +- Everywhere else, use `bash` if possible. It's more powerful than `sh` but still a widespread shell. + +## Code style and format + +This section describes the tools that should be made a mandatory part of +a project's CI pipeline if it contains shell scripts. These tools +automate shell code formatting, checking for errors or vulnerabilities, etc. + +### Linting + +We're using the [ShellCheck](https://www.shellcheck.net/) utility in its default configuration to lint our +shell scripts. + +All projects with shell scripts should use this GitLab CI/CD job: + +```yaml +shell check: + image: koalaman/shellcheck-alpine + stage: test + before_script: + - shellcheck --version + script: + - shellcheck scripts/**/*.sh # path to your shell scripts +``` + +TIP: **Tip:** +By default, ShellCheck will use the [shell detection](https://github.com/koalaman/shellcheck/wiki/SC2148#rationale) +to determine the shell dialect in use. If the shell file is out of your control and ShellCheck cannot +detect the dialect, use `-s` flag to specify it: `-s sh` or `-s bash`. + +### Formatting + +It's recommended to use the [shfmt](https://github.com/mvdan/sh#shfmt) tool to maintain consistent formatting. +We format shell scripts according to the [Google Shell Style Guide](https://google.github.io/styleguide/shell.xml), +so the following `shfmt` invocation should be applied to the project's script files: + +```bash +shfmt -i 2 -ci scripts/**/*.sh +``` + +TIP: **Tip:** +By default, shfmt will use the [shell detection](https://github.com/mvdan/sh#shfmt) similar to one of ShellCheck +and ignore files starting with a period. To override this, use `-ln` flag to specify the shell dialect: +`-ln posix` or `-ln bash`. + +NOTE: **Note:** +Currently, the `shfmt` tool [is not shipped](https://github.com/mvdan/sh/issues/68) as a Docker image containing +a Linux shell. This makes it impossible to use the [official Docker image](https://hub.docker.com/r/mvdan/shfmt) +in GitLab Runner. This [may change](https://github.com/mvdan/sh/issues/68#issuecomment-507721371) in future. + +## Testing + +NOTE: **Note:** +This is a work in progress. + +It is an [ongoing effort](https://gitlab.com/gitlab-org/gitlab-ce/issues/64016) to evaluate different tools for the +automated testing of shell scripts (like [BATS](https://github.com/sstephenson/bats)). + +## Code Review + +The code review should be performed according to: + +- [ShellCheck Checks list](https://github.com/koalaman/shellcheck/wiki/Checks) +- [Google Shell Style Guide](https://google.github.io/styleguide/shell.xml) +- [Shfmt formatting caveats](https://github.com/mvdan/sh#caveats) + +However, the recommended course of action is to use the aforementioned +tools and address reported offenses. This should eliminate the need +for code review. + +--- + +[Return to Development documentation](../README.md). diff --git a/doc/development/sql.md b/doc/development/sql.md index a256fd46c09..2584dcfb4ca 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -15,14 +15,11 @@ FROM issues WHERE title LIKE 'WIP:%'; ``` -On PostgreSQL the `LIKE` statement is case-sensitive. On MySQL this depends on -the case-sensitivity of the collation, which is usually case-insensitive. To -perform a case-insensitive `LIKE` on PostgreSQL you have to use `ILIKE` instead. -This statement in turn isn't supported on MySQL. +On PostgreSQL the `LIKE` statement is case-sensitive. To perform a case-insensitive +`LIKE` you have to use `ILIKE` instead. -To work around this problem you should write `LIKE` queries using Arel instead -of raw SQL fragments as Arel automatically uses `ILIKE` on PostgreSQL and `LIKE` -on MySQL. This means that instead of this: +To handle this automatically you should use `LIKE` queries using Arel instead +of raw SQL fragments, as Arel automatically uses `ILIKE` on PostgreSQL. ```ruby Issue.where('title LIKE ?', 'WIP:%') @@ -45,7 +42,7 @@ table = Issue.arel_table Issue.where(table[:title].matches('WIP:%').or(table[:foo].matches('WIP:%'))) ``` -For PostgreSQL this produces: +On PostgreSQL, this produces: ```sql SELECT * @@ -53,18 +50,10 @@ FROM issues WHERE (title ILIKE 'WIP:%' OR foo ILIKE 'WIP:%') ``` -In turn for MySQL this produces: - -```sql -SELECT * -FROM issues -WHERE (title LIKE 'WIP:%' OR foo LIKE 'WIP:%') -``` - ## LIKE & Indexes -Neither PostgreSQL nor MySQL use any indexes when using `LIKE` / `ILIKE` with a -wildcard at the start. For example, this will not use any indexes: +PostgreSQL won't use any indexes when using `LIKE` / `ILIKE` with a wildcard at +the start. For example, this will not use any indexes: ```sql SELECT * @@ -75,9 +64,8 @@ WHERE title ILIKE '%WIP:%'; Because the value for `ILIKE` starts with a wildcard the database is not able to use an index as it doesn't know where to start scanning the indexes. -MySQL provides no known solution to this problem. Luckily PostgreSQL _does_ -provide a solution: trigram GIN indexes. These indexes can be created as -follows: +Luckily, PostgreSQL _does_ provide a solution: trigram GIN indexes. These +indexes can be created as follows: ```sql CREATE INDEX [CONCURRENTLY] index_name_here diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 448d9fd01c4..f30a83a4c71 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -15,16 +15,6 @@ manifest themselves within our code. When designing our tests, take time to revi our test design. We can find some helpful heuristics documented in the Handbook in the [Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section. -## Run tests against MySQL - -By default, tests are only run against PostgreSQL, but you can run them on -demand against MySQL by following one of the following conventions: - -| Convention | Valid example | -|:----------------------|:-----------------------------| -| Include `mysql` in your branch name | `enhance-mysql-support` | -| Include `[run mysql]` in your commit message | `Fix MySQL support<br><br>[run mysql]` | - ## Test speed GitLab has a massive test suite that, without [parallelization], can take hours @@ -70,6 +60,7 @@ bundle exec rspec spec/[path]/[to]/[spec].rb - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, use a Capyabara matcher beforehand (e.g. `find('.js-foo')`) to ensure the element actually exists. +- Use `focus: true` to isolate parts of the specs you want to run. [four-phase-test]: https://robots.thoughtbot.com/four-phase-test @@ -454,6 +445,19 @@ complexity of RSpec expectations.They should be placed under a certain type of specs only (e.g. features, requests etc.) but shouldn't be if they apply to multiple type of specs. +#### `be_like_time` + +Time returned from a database can differ in precision from time objects +in Ruby, so we need flexible tolerances when comparing in specs. We can +use `be_like_time` to compare that times are within one second of each +other. + +Example: + +```ruby +expect(metrics.merged_at).to be_like_time(time) +``` + #### `have_gitlab_http_status` Prefer `have_gitlab_http_status` over `have_http_status` because the former diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 87d48726268..d9f66a827de 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -39,7 +39,6 @@ slowest test files and try to improve them. ## CI setup -- On CE and EE, the test suite runs both PostgreSQL and MySQL. - Rails logging to `log/test.log` is disabled by default in CI [for performance reasons][logging]. To override this setting, provide the `RAILS_ENABLE_TEST_LOG` environment variable. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 59eb3ecfd7e..3ae3ce183d9 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -45,11 +45,11 @@ Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests -#### Using the `package-and-qa` job +#### Using the `package-and-qa-manual` job It is possible to run end-to-end tests for a merge request, eventually being run in a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project, -by triggering the `package-and-qa` manual action in the `test` stage (not +by triggering the `package-and-qa-manual` manual action in the `test` stage (not available for forks). **This runs end-to-end tests against a custom Omnibus package built from your @@ -65,28 +65,23 @@ Below you can read more about how to use it and how does it work. Currently, we are using _multi-project pipeline_-like approach to run QA pipelines. - - -<details> -<summary>Show mermaid source</summary> -<pre> +```mermaid graph LR A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2 - B2[<b>`Trigger-qa` stage</b><br />`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 + B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 -subgraph gitlab-ce/ee pipeline - A1[<b>`test` stage</b><br />`package-and-qa` job] +subgraph "gitlab-ce/ee pipeline" + A1[`test` stage<br>`package-and-qa-manual` job] end -subgraph omnibus-gitlab pipeline - A2[<b>`Trigger-docker` stage</b><br />`Trigger:gitlab-docker` job] -->|once done| B2 +subgraph "omnibus-gitlab pipeline" + A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end -subgraph gitlab-qa pipeline - A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br />and post the result on the original commit tested| A1 +subgraph "gitlab-qa pipeline" + A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa-manual` job<br>and post the result on the original commit tested| A1 end -</pre> -</details> +``` 1. Developer triggers a manual action, that can be found in CE / EE merge requests. This starts a chain of pipelines in multiple projects. @@ -148,7 +143,7 @@ Once you decided where to put [test environment orchestration scenarios] and the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing instance-level scenarios][instance-level scenarios]. -Continued reading: +Continued reading: - [Quick Start Guide](quick_start_guide.md) - [Style Guide](style_guide.md) diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 05cb03eb4bd..850ea6b60ac 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -27,7 +27,7 @@ When someone later changes `t.text_field :login` in the view associated with this page to `t.text_field :username` it will generate a different field identifier, what would effectively break all tests. -Because we are using `Page::Main::Login.act { sign_in_using_credentials }` +Because we are using `Page::Main::Login.perform(&:sign_in_using_credentials)` everywhere, when we want to sign into GitLab, the page object is the single source of truth, and we will need to update `fill_in :user_login` to `fill_in :user_username` only in a one place. @@ -40,7 +40,7 @@ the time it would take to build packages and test everything. That is why when someone changes `t.text_field :login` to `t.text_field :username` in the _new session_ view we won't know about this change until our GitLab QA nightly pipeline fails, or until someone triggers -`package-and-qa` action in their merge request. +`package-and-qa-manual` action in their merge request. Obviously such a change would break all tests. We call this problem a _fragile tests problem_. @@ -92,20 +92,25 @@ end The `view` DSL method will correspond to the rails View, partial, or vue component that renders the elements. The `element` DSL method in turn declares an element for which a corresponding -`qa-element-name-dasherized` CSS class will need to be added to the view file. +`data-qa-selector=element_name_snaked` data attribute will need to be added to the view file. You can also define a value (String or Regexp) to match to the actual view code but **this is deprecated** in favor of the above method for two reasons: - Consistency: there is only one way to define an element -- Separation of concerns: QA uses dedicated CSS classes instead of reusing code +- Separation of concerns: QA uses dedicated `data-qa-*` attributes instead of reusing code or classes used by other components (e.g. `js-*` classes etc.) ```ruby view 'app/views/my/view.html.haml' do - # Implicitly require `.qa-logout-button` CSS class to be present in the view + + ### Good ### + + # Implicitly require the CSS selector `[data-qa-selector="logout_button"]` to be present in the view element :logout_button + ### Bad ### + ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. # Require `f.submit "Sign in"` to be present in `my/view.html.haml element :my_button, 'f.submit "Sign in"' # rubocop:disable QA/ElementWithPattern @@ -129,24 +134,38 @@ view 'app/views/my/view.html.haml' do end ``` -To add these elements to the view, you must change the rails View, partial, or vue component by adding a `qa-element-descriptor` class +To add these elements to the view, you must change the rails View, partial, or vue component by adding a `data-qa-selector` attribute for each element defined. -In our case, `qa-login-field`, `qa-password-field` and `qa-sign-in-button` +In our case, `data-qa-selector="login_field"`, `data-qa-selector="password_field"` and `data-qa-selector="sign_in_button"` **app/views/my/view.html.haml** ```haml -= f.text_field :login, class: "form-control top qa-login-field", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required." -= f.password_field :password, class: "form-control bottom qa-password-field", required: true, title: "This field is required." -= f.submit "Sign in", class: "btn btn-success qa-sign-in-button" += f.text_field :login, class: "form-control top", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required.", data: { qa_selector: 'login_field' } += f.password_field :password, class: "form-control bottom", required: true, title: "This field is required.", data: { qa_selector: 'password_field' } += f.submit "Sign in", class: "btn btn-success", data: { qa_selector: 'sign_in_button' } ``` Things to note: -- The CSS class must be `kebab-cased` (separated with hyphens "`-`") +- The name of the element and the qa_selector must match and be snake_cased - If the element appears on the page unconditionally, add `required: true` to the element. See [Dynamic element validation](dynamic_element_validation.md) +- You may see `.qa-selector` classes in existing Page Objects. We should prefer the [`data-qa-selector`](#data-qa-selector-vs-qa-selector) + method of definition over the `.qa-selector` CSS class + +### `data-qa-selector` vs `.qa-selector` + +> Introduced in GitLab 12.1 + +There are two supported methods of defining elements within a view. + +1. `data-qa-selector` attribute +1. `.qa-selector` class + +Any existing `.qa-selector` class should be considered deprecated +and we should prefer the `data-qa-selector` method of definition. ## Running the test locally diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md index 064fb0e31dd..e1df8be8b6f 100644 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -101,7 +101,7 @@ it 'replaces an existing label if it has the same key' do page.find('#content-body').click page.refresh - labels_block = page.find('.qa-labels-block') + labels_block = page.find(%q([data-qa-selector="labels_block"])) expect(labels_block).to have_content('animal::dolphin') expect(labels_block).not_to have_content('animal::fox') @@ -110,15 +110,15 @@ end ``` > Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. - +> > The exemplified test case's MVC is not enough for the change to be merged, but it helps to build up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects] before they can be merged. This way we better separate the responsibilities, where the Page Objects encapsulate elements and methods that allow us to interact with pages, while the spec files describe the test cases in more business-related language. Below are the steps that the test covers: 1. The test finds the 'Edit' link for the labels and clicks on it. -2. Then it fills in the 'Assign labels' input field with the value 'animal::dolphin' and press enters. -3. Then it clicks in the content body to apply the label and refreshes the page. -4. Finally, the expectations check that the previous scoped label was removed and that the new one was added. +1. Then it fills in the 'Assign labels' input field with the value 'animal::dolphin' and press enters. +1. Then it clicks in the content body to apply the label and refreshes the page. +1. Finally, the expectations check that the previous scoped label was removed and that the new one was added. Let's now see how the second test case would look. @@ -130,7 +130,7 @@ it 'keeps both scoped labels when adding a label with a different key' do page.find('#content-body').click page.refresh - labels_block = page.find('.qa-labels-block') + labels_block = page.find(%q([data-qa-selector="labels_block"])) expect(labels_block).to have_content('animal::fox') expect(labels_block).to have_content('plant::orchid') @@ -139,14 +139,14 @@ it 'keeps both scoped labels when adding a label with a different key' do end ``` -> Note that elements are always located using CSS selectors, and a good practice is to add test-specific selectors (this is called adding testability to the application and we will talk more about it later.) For example, the `labels_block` element uses the selector `.qa-labels-block`, which was added specifically for testing purposes. +> Note that elements are always located using CSS selectors, and a good practice is to add test-specific selectors (this is called "testability"). For example, the `labels_block` element uses the CSS selector [`data-qa-selector="labels_block"`](page_objects.md#data-qa-selector-vs-qa-selector), which was added specifically for testing purposes. Below are the steps that the test covers: 1. The test finds the 'Edit' link for the labels and clicks on it. -2. Then it fills in the 'Assign labels' input field with the value 'plant::orchid' and press enters. -3. Then it clicks in the content body to apply the label and refreshes the page. -4. Finally, the expectations check that both scoped labels are present. +1. Then it fills in the 'Assign labels' input field with the value 'plant::orchid' and press enters. +1. Then it clicks in the content body to apply the label and refreshes the page. +1. Finally, the expectations check that both scoped labels are present. > Similar to the previous test, this one is also very straightforward, but there is some code duplication. Let's address it. @@ -168,7 +168,7 @@ end it 'replaces an existing label if it has the same key' do select_label_and_refresh @new_label_same_scope - labels_block = page.find('.qa-labels-block') + labels_block = page.find(%q([data-qa-selector="labels_block"])) expect(labels_block).to have_content(@new_label_same_scope) expect(labels_block).not_to have_content(@initial_label) @@ -179,7 +179,7 @@ end it 'keeps both scoped label when adding a label with a different key' do select_label_and_refresh @new_label_different_scope - labels_block = page.find('.qa-labels-block') + labels_block = page.find(%q([data-qa-selector="labels_block"])) expect(labels_blocks).to have_content(@new_label_different_scope) expect(labels_blocks).to have_content(@initial_label) @@ -211,7 +211,7 @@ A pre-condition for the entire test suite is defined in the `before :context` bl > For our test suite, due to the need of the tests being completely independent of each other, we won't use the `before :context` block. The `before :context` block would make the tests dependent on each other because the first test changes the label of the issue, and the second one depends on the `'animal::fox'` label being set. -> **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. +TIP: **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. #### `before` @@ -222,7 +222,7 @@ As the pre-conditions for our test suite, the things that needs to happen before - A project being created with an issue and labels already set; - The issue page being opened with only one scoped label applied to it. -> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-remote-grid-environment-variables), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). +> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). #### Implementation @@ -274,11 +274,11 @@ end In the `before` block we create all the application state needed for the tests to run. We do that by using the `Runtime::Browser.visit` method to go to the login page, by performing a `sign_in_using_credentials` from the `Login` Page Object, by fabricating resources via APIs (`issue`, and `Resource::Label`), and by using the `issue.visit!` to visit the issue page. > A project is created in the background by creating the `issue` resource. - +> > When creating the [Resources], notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `title`, and `labels` for the `issue` resource; and `project` and `title` for the `label` resource. - +> > What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. - +> > Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. ### 6. Optimization @@ -290,7 +290,7 @@ As already mentioned in the [best practices](best_practices.md) document, end-to Some improvements that we could make in our test suite to optimize its time to run are: 1. Having a single test case (an `it` block) that exercises both scenarios to avoid "wasting" time in the tests' pre-conditions, instead of having two different test cases. -2. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. +1. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. Let's look at a suggestion that addresses the above points, one by one: @@ -305,7 +305,7 @@ module QA it 'correctly applies scoped labels depending on if they are from the same or a different scope' do select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] - labels_block = page.all('.qa-labels-block') + labels_block = page.all(%q([data-qa-selector="labels_block"])) expect(labels_block).to have_content(@new_label_same_scope) expect(labels_block).to have_content(@new_label_different_scope) @@ -332,8 +332,8 @@ To address point 1, we changed the test implementation from two `it` blocks into > Notice that the implementation of the new and unique `it` block had to change a little bit. Below we describe in details what it does. 1. It selects two scoped labels simultaneously, one from the same scope of the one already applied in the issue during the setup phase (in the `before` block), and another one from a different scope. -2. It asserts that the correct labels are visible in the `labels_block`, and that the labels were correctly added and removed; -3. Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) +1. It asserts that the correct labels are visible in the `labels_block`, and that the labels were correctly added and removed; +1. Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) ### 7. Resources @@ -362,7 +362,7 @@ First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d358 Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). > This line is needed to allow for the issue fabrication, and for labels to be automatically added to the issue when fabricating it via API. - +> > We add the attributes above the existing attribute to keep them alphabetically organized. Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: @@ -437,7 +437,7 @@ By defining the `resource_web_url(resource)` method, we override the one from th By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. -By defining the `api_post_path` method, we allow for the [`ApiFabricator `](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. +By defining the `api_post_path` method, we allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. @@ -542,9 +542,9 @@ end Notice that we have not only moved the `select_labels_and_refresh` method, but we have also changed its implementation to: 1. Click the `:edit_link_labels` element previously defined, instead of using `find('.block.labels .edit-link').click` -2. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` -3. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` -4. Iterate on every label again, and then we use `has_element?(:labels_block, text: label)` after clicking the page body (which applies the labels), and before refreshing the page, to avoid test flakiness due to refreshing too fast. +1. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` +1. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` +1. Iterate on every label again, and then we use `has_element?(:labels_block, text: label)` after clicking the page body (which applies the labels), and before refreshing the page, to avoid test flakiness due to refreshing too fast. ##### Details of `text_of_labels_block` @@ -552,37 +552,36 @@ The `text_of_labels_block` method is a simple method that returns the `:labels_b #### Updates in the view (*.html.haml) and `dropdowns_helper.rb` files -Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the Page Object. +Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the [Page Objects]. -In the [app/views/shared/issuable/_sidebar.html.haml](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/views/shared/issuable/_sidebar.html.haml) file, on [line 105 ](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L105), add an extra class `qa-edit-link-labels`. +In [`app/views/shared/issuable/_sidebar.html.haml:105`](https://gitlab.com/gitlab-org/gitlab-ee/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L105), add a `data: { qa_selector: 'edit_link_labels' }` data attribute. The code should look like this: ```haml -= link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link float-right qa-edit-link-labels' += link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link float-right', data: { qa_selector: 'edit_link_labels' } ``` -In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L121), add an extra class `.qa-dropdown-menu-labels`. +In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab-ee/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L121), add a `data: { qa_selector: 'dropdown_menu_labels' }` data attribute. The code should look like this: ```haml -.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.dropdown-menu-selectable.qa-dropdown-menu-labels +.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.dropdown-menu-selectable.dropdown-extended-height{ data: { qa_selector: 'dropdown_menu_labels' } } ``` -In the [`dropdowns_helper.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/helpers/dropdowns_helper.rb) file, on [line 94](https://gitlab.com/gitlab-org/gitlab-ee/blob/99e51a374f2c20bee0989cac802e4b5621f72714/app/helpers/dropdowns_helper.rb#L94), add an extra class `qa-dropdown-input-field`. +In [`app/helpers/dropdowns_helper.rb:94`](https://gitlab.com/gitlab-org/gitlab-ee/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/helpers/dropdowns_helper.rb#L94), add a `data: { qa_selector: 'dropdown_input_field' }` data attribute. The code should look like this: ```ruby -filter_output = search_field_tag search_id, nil, class: "dropdown-input-field qa-dropdown-input-field", placeholder: placeholder, autocomplete: 'off' +filter_output = search_field_tag search_id, nil, class: "dropdown-input-field", placeholder: placeholder, autocomplete: 'off', data: { qa_selector: 'dropdown_input_field' } ``` -> Classes starting with `qa-` are used for testing purposes only, and by defining such classes in the elements we add **testability** in the application. - -> When defining a class like `qa-labels-block`, it is transformed into `:labels_block` for usage in the Page Objects. So, `qa-edit-link-labels` is transformed into `:edit_link_labels`, `qa-dropdown-menu-labels` is transformed into `:dropdown_menu_labels`, and `qa-dropdown-input-field` is transformed into `:dropdown_input_field`. Also, we use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective `qa-` selectors in the specified views. - -> We did not define the `qa-labels-block` class in the `app/views/shared/issuable/_sidebar.html.haml` file because it was already there to be used. +> `data-qa-*` data attributes and CSS classes starting with `qa-` are used solely for the purpose of QA and testing. +> By defining these, we add **testability** to the application. +> +> When defining a data attribute like: `qa_selector: 'labels_block'`, it should match the element definition: `element :labels_block`. We use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective selectors in the specified views. #### Updates in the `QA::Page::Base` class @@ -600,8 +599,6 @@ This method receives an element (`name`) and the `keys` that it will send to tha As you might remember, in the Issue Page Object we call this method like this: `send_keys_to_element(:dropdown_input_field, [label, :enter])`. -___ - With that, you should be able to start writing end-to-end tests yourself. *Congratulations!* [Page Objects]: page_objects.md diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 0272e1810f2..97560e616a1 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -45,7 +45,7 @@ Notice that in the above example, before clicking the `:operations_environments_ > We can create these methods as helpers to abstract multi-step navigation. -### Element naming convention +## Element naming convention When adding new elements to a page, it's important that we have a uniform element naming convention. @@ -63,17 +63,17 @@ We follow a simple formula roughly based on hungarian notation. - `_checkbox` - `_radio` - `_content` - + *Note: This list is a work in progress. This list will eventually be the end-all enumeration of all available types. I.e., any element that does not end with something in this list is bad form.* - -#### Examples + +### Examples **Good** ```ruby view '...' do - element :edit_button + element :edit_button element :notes_tab element :squash_checkbox element :username_field @@ -84,16 +84,61 @@ end **Bad** ```ruby -view '...' do +view '...' do # `_confirmation` should be `_field`. what sort of confirmation? a checkbox confirmation? no real way to disambiguate. # an appropriate replacement would be `element :password_confirmation_field` element :password_confirmation - # `clone_options` is too vague. If it's a dropdown menu, it should be `clone_dropdown`. + # `clone_options` is too vague. If it's a dropdown menu, it should be `clone_dropdown`. # If it's a checkbox, it should be `clone_checkbox` element :clone_options - + # how is this url being displayed? is it a textbox? a simple span? + # If it is content on the page, it should be `ssh_clone_url_content` element :ssh_clone_url end ``` + +## Block argument naming + +To have a standard on how we call pages when using the `.perform` method, we use the name of page object being called, all lowercased, and separated by underscore, if needed (see good and bad examples below.) This also applies to resources. We chose not to simply use `page` because that would shadow the Capybara DSL, potentially leading to confusion and bugs. + +### Examples + +**Good** + +```ruby +# qa/specs/features/browser_ui/1_manage/project/add_project_member_spec.rb + +Page::Project::Settings::Members.perform do |members| + members.do_something +end +``` + +```ruby +# qa/specs/features/ee/browser_ui/3_create/merge_request/add_batch_comments_in_merge_request_spec.rb + +Resource::MergeRequest.fabricate! do |merge_request| + merge_request.do_something_else +end +``` + +**Bad** + +```ruby +# qa/specs/features/browser_ui/1_manage/project/add_project_member_spec.rb + +Page::Project::Settings::Members.perform do |project_settings_members_page| + project_settings_members_page.do_something +end +``` + +```ruby +# qa/specs/features/ee/browser_ui/3_create/merge_request/add_batch_comments_in_merge_request_spec.rb + +Resource::MergeRequest.fabricate! do |merge_request_page| + merge_request_page.do_something_else +end +``` + +> Besides the advantage of having a standard in place, by following this standard we also write shorter lines of code.
\ No newline at end of file diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 931cbc51cae..eb0bf6fc563 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -35,8 +35,8 @@ Once a test is in quarantine, there are 3 choices: Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: -- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE) -- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only) +- `rspec-pg-quarantine` (CE & EE) +- `rspec-pg-quarantine-ee` (EE only) ## Automatic retries and flaky tests detection diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 98df0b5ea7c..7dc89a3fcdb 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -79,6 +79,37 @@ describe('Component', () => { Remember that the performance of each test depends on the environment. +### Manual module mocks + +Jest supports [manual module mocks](https://jestjs.io/docs/en/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module. **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder), and the logic that Jest uses to apply mocks from `__mocks__/` is rather inconsistent. + +Instead, our test runner detects manual mocks from `spec/frontend/mocks/`. Any mock placed here is automatically picked up and injected whenever you import its source module. + +- Files in `spec/frontend/mocks/ce` will mock the corresponding CE module from `app/assets/javascripts`, mirroring the source module's path. + - Example: `spec/frontend/mocks/ce/lib/utils/axios_utils` will mock the module `~/lib/utils/axios_utils`. +- Files in `spec/frontend/mocks/node` will mock NPM packages of the same name or path. +- We don't support mocking EE modules yet. + +If a mock is found for which a source module doesn't exist, the test suite will fail. 'Virtual' mocks, or mocks that don't have a 1-to-1 association with a source module, are not supported yet. + +#### Writing a mock + +Create a JS module in the appropriate place in `spec/frontend/mocks/`. That's it. It will automatically mock its source package in all tests. + +Make sure that your mock's export has the same format as the mocked module. So, if you're mocking a CommonJS module, you'll need to use `module.exports` instead of the ES6 `export`. + +It might be useful for a mock to expose a property that indicates if the mock was loaded. This way, tests can assert the presence of a mock without calling any logic and causing side-effects. The `~/lib/utils/axios_utils` module mock has such a property, `isMock`, that is `true` in the mock and undefined in the original class. Jest's mock functions also have a `mock` property that you can test. + +#### Bypassing mocks + +If you ever need to import the original module in your tests, use [`jest.requireActual()`](https://jestjs.io/docs/en/jest-object#jestrequireactualmodulename) (or `jest.requireActual().default` for the default export). The `jest.mock()` and `jest.unmock()` won't have an effect on modules that have a manual mock, because mocks are imported and cached before any tests are run. + +#### Keep mocks light + +Global mocks introduce magic and can affect how modules are imported in your tests. Try to keep them as light as possible and dependency-free. A global mock should be useful for any unit test. For example, the `axios_utils` and `jquery` module mocks throw an error when an HTTP request is attempted, since this is useful behaviour in >99% of tests. + +When in doubt, construct mocks in your test file using [`jest.mock()`](https://jestjs.io/docs/en/jest-object#jestmockmodulename-factory-options), [`jest.spyOn()`](https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname), etc. + ## Karma test suite GitLab uses the [Karma][karma] test runner with [Jasmine] as its test @@ -201,7 +232,7 @@ module. GitLab has a custom `spyOnDependency` method which utilizes [babel-plugin-rewire](https://github.com/speedskater/babel-plugin-rewire) to achieve this. It can be used like so: -```js +```javascript // my_module.js import { visitUrl } from '~/lib/utils/url_utility'; @@ -210,7 +241,7 @@ export default function doSomething() { } ``` -```js +```javascript // my_module_spec.js import doSomething from '~/my_module'; @@ -434,7 +465,7 @@ See this [section][vue-test]. For running the frontend tests, you need the following commands: -- `rake karma:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). +- `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). - `yarn test` executes the tests. As long as the fixtures don't change, `yarn test` is sufficient (and saves you some time). @@ -487,8 +518,8 @@ Information on setting up and running RSpec integration tests with Code that is added to HAML templates (in `app/views/`) or makes Ajax requests to the backend has tests that require HTML or JSON from the backend. Fixtures for these tests are located at: -- `spec/javascripts/fixtures/`, for running tests in CE. -- `ee/spec/javascripts/fixtures/`, for running tests in EE. +- `spec/frontend/fixtures/`, for running tests in CE. +- `ee/spec/frontend/fixtures/`, for running tests in EE. Fixture files in: @@ -499,29 +530,29 @@ The following are examples of tests that work for both Karma and Jest: ```javascript it('makes a request', () => { - const responseBody = getJSONFixture('some/fixture.json'); // loads spec/javascripts/fixtures/some/fixture.json + const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json axiosMock.onGet(endpoint).reply(200, responseBody); - + myButton.click(); - + // ... }); it('uses some HTML element', () => { - loadFixtures('some/page.html'); // loads spec/javascripts/fixtures/some/page.html and adds it to the DOM - + loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM + const element = document.getElementById('#my-id'); - + // ... }); ``` -HTML and JSON fixtures are generated from backend views and controllers using RSpec (see `spec/javascripts/fixtures/*.rb`). +HTML and JSON fixtures are generated from backend views and controllers using RSpec (see `spec/frontend/fixtures/*.rb`). For each fixture, the content of the `response` variable is stored in the output file. This variable gets automagically set if the test is marked as `type: :request` or `type: :controller`. -Fixtures are regenerated using the `bin/rake karma:fixtures` command but you can also generate them individually, -for example `bin/rspec spec/javascripts/fixtures/merge_requests.rb`. +Fixtures are regenerated using the `bin/rake frontend:fixtures` command but you can also generate them individually, +for example `bin/rspec spec/frontend/fixtures/merge_requests.rb`. When creating a new fixture, it often makes sense to take a look at the corresponding tests for the endpoint in `(ee/)spec/controllers/` or `(ee/)spec/requests/`. ## Gotchas @@ -557,11 +588,522 @@ end [jasmine-focus]: https://jasmine.github.io/2.5/focused_specs.html [karma]: http://karma-runner.github.io/ -[vue-test]: https://docs.gitlab.com/ce/development/fe_guide/vue.html#testing-vue-components +[vue-test]: ../fe_guide/vue.md#testing-vue-components [rspec]: https://github.com/rspec/rspec-rails#feature-specs [capybara]: https://github.com/teamcapybara/capybara [jasmine]: https://jasmine.github.io/ +## Overview of Frontend Testing Levels + +Tests relevant for frontend development can be found at the following places: + +- `spec/javascripts/` which are run by Karma (command: `yarn karma`) and contain + - [frontend unit tests](#frontend-unit-tests) + - [frontend component tests](#frontend-component-tests) + - [frontend integration tests](#frontend-integration-tests) +- `spec/frontend/` which are run by Jest (command: `yarn jest`) and contain + - [frontend unit tests](#frontend-unit-tests) + - [frontend component tests](#frontend-component-tests) + - [frontend integration tests](#frontend-integration-tests) +- `spec/features/` which are run by RSpec and contain + - [feature tests](#feature-tests) + +All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-ce/issues/52483)). + +In addition, there used to be feature tests in `features/`, run by Spinach. +These were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-ce/issues/23036)). + +See also [Notes on testing Vue components](../fe_guide/vue.html#testing-vue-components). + +### Frontend unit tests + +Unit tests are on the lowest abstraction level and typically test functionality that is not directly perceivable by a user. + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class plain tested; + class Vuex tested; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use unit tests + +<details> + <summary>exported functions and classes</summary> + Anything that is exported can be reused at various places in a way you have no control over. + Therefore it is necessary to document the expected behavior of the public interface with tests. +</details> + +<details> + <summary>Vuex actions</summary> + Any Vuex action needs to work in a consistent way independent of the component it is triggered from. +</details> + +<details> + <summary>Vuex mutations</summary> + For complex Vuex mutations it helps to identify the source of a problem by separating the tests from other parts of the Vuex store. +</details> + +#### When *not* to use unit tests + +<details> + <summary>non-exported functions or classes</summary> + Anything that is not exported from a module can be considered private or an implementation detail and doesn't need to be tested. +</details> + +<details> + <summary>constants</summary> + Testing the value of a constant would mean to copy it. + This results in extra effort without additional confidence that the value is correct. +</details> + +<details> + <summary>Vue components</summary> + Computed properties, methods, and lifecycle hooks can be considered an implementation detail of components and don't need to be tested. + They are implicitly covered by component tests. + The <a href="https://vue-test-utils.vuejs.org/guides/#getting-started">official Vue guidelines</a> suggest the same. +</details> + +#### What to mock in unit tests + +<details> + <summary>state of the class under test</summary> + Modifying the state of the class under test directly rather than using methods of the class avoids side-effects in test setup. +</details> + +<details> + <summary>other exported classes</summary> + Every class needs to be tested in isolation to prevent test scenarios from growing exponentially. +</details> + +<details> + <summary>single DOM elements if passed as parameters</summary> + For tests that only operate on single DOM elements rather than a whole page, creating these elements is cheaper than loading a whole HTML fixture. +</details> + +<details> + <summary>all server requests</summary> + When running frontend unit tests, the backend may not be reachable. + Therefore all outgoing requests need to be mocked. +</details> + +<details> + <summary>asynchronous background operations</summary> + Background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. +</details> + +#### What *not* to mock in unit tests + +<details> + <summary>non-exported functions or classes</summary> + Everything that is not exported can be considered private to the module and will be implicitly tested via the exported classes / functions. +</details> + +<details> + <summary>methods of the class under test</summary> + By mocking methods of the class under test, the mocks will be tested and not the real methods. +</details> + +<details> + <summary>utility functions (pure functions, or those that only modify parameters)</summary> + If a function has no side effects because it has no state, it is safe to not mock it in tests. +</details> + +<details> + <summary>full HTML pages</summary> + Loading the HTML of a full page slows down tests, so it should be avoided in unit tests. +</details> + +### Frontend component tests + +Component tests cover the state of a single component that is perceivable by a user depending on external signals such as user input, events fired from other components, or application state. + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class Vue tested; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use component tests + +- Vue components + +#### When *not* to use component tests + +<details> + <summary>Vue applications</summary> + Vue applications may contain many components. + Testing them on a component level requires too much effort. + Therefore they are tested on frontend integration level. +</details> + +<details> + <summary>HAML templates</summary> + HAML templates contain only Markup and no frontend-side logic. + Therefore they are not complete components. +</details> + +#### What to mock in component tests + +<details> + <summary>DOM</summary> + Operating on the real DOM is significantly slower than on the virtual DOM. +</details> + +<details> + <summary>properties and state of the component under test</summary> + Similarly to testing classes, modifying the properties directly (rather than relying on methods of the component) avoids side-effects. +</details> + +<details> + <summary>Vuex store</summary> + To avoid side effects and keep component tests simple, Vuex stores are replaced with mocks. +</details> + +<details> + <summary>all server requests</summary> + Similar to unit tests, when running component tests, the backend may not be reachable. + Therefore all outgoing requests need to be mocked. +</details> + +<details> + <summary>asynchronous background operations</summary> + Similar to unit tests, background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. +</details> + +<details> + <summary>child components</summary> + Every component is tested individually, so child components are mocked. + See also <a href="https://vue-test-utils.vuejs.org/api/#shallowmount">shallowMount()</a> +</details> + +#### What *not* to mock in component tests + +<details> + <summary>methods or computed properties of the component under test</summary> + By mocking part of the component under test, the mocks will be tested and not the real component. +</details> + +<details> + <summary>functions and classes independent from Vue</summary> + All plain JavaScript code is already covered by unit tests and needs not to be mocked in component tests. +</details> + +### Frontend integration tests + +Integration tests cover the interaction between all components on a single page. +Their abstraction level is comparable to how a user would interact with the UI. + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class plain tested; + class Vue tested; + class Vuex tested; + class GraphQL tested; + class browser tested; + linkStyle 0,1,2,3,4,5,6 stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use integration tests + +<details> + <summary>page bundles (<code>index.js</code> files in <code>app/assets/javascripts/pages/</code>)</summary> + Testing the page bundles ensures the corresponding frontend components integrate well. +</details> + +<details> + <summary>Vue applications outside of page bundles</summary> + Testing Vue applications as a whole ensures the corresponding frontend components integrate well. +</details> + +#### What to mock in integration tests + +<details> + <summary>HAML views (use fixtures instead)</summary> + Rendering HAML views requires a Rails environment including a running database which we cannot rely on in frontend tests. +</details> + +<details> + <summary>all server requests</summary> + Similar to unit and component tests, when running component tests, the backend may not be reachable. + Therefore all outgoing requests need to be mocked. +</details> + +<details> + <summary>asynchronous background operations that are not perceivable on the page</summary> + Background operations that affect the page need to be tested on this level. + All other background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. +</details> + +#### What *not* to mock in integration tests + +<details> + <summary>DOM</summary> + Testing on the real DOM ensures our components work in the environment they are meant for. + Part of this will be delegated to <a href="https://gitlab.com/gitlab-org/quality/team-tasks/issues/45">cross-browser testing</a>. +</details> + +<details> + <summary>properties or state of components</summary> + On this level, all tests can only perform actions a user would do. + For example to change the state of a component, a click event would be fired. +</details> + +<details> + <summary>Vuex stores</summary> + When testing the frontend code of a page as a whole, the interaction between Vue components and Vuex stores is covered as well. +</details> + +### Feature tests + +In contrast to [frontend integration tests](#frontend-integration-tests), feature tests make requests against the real backend instead of using fixtures. +This also implies that database queries are executed which makes this category significantly slower. + +See also the [RSpec testing guidelines](../testing_guide/best_practices.md#rspec). + +```mermaid +graph RL + plain[Plain JavaScript]; + Vue[Vue Components]; + feature-flags[Feature Flags]; + license-checks[License Checks]; + + plain---Vuex; + plain---GraphQL; + Vue---plain; + Vue---Vuex; + Vue---GraphQL; + browser---plain; + browser---Vue; + plain---backend; + Vuex---backend; + GraphQL---backend; + Vue---backend; + backend---database; + backend---feature-flags; + backend---license-checks; + + class backend tested; + class plain tested; + class Vue tested; + class Vuex tested; + class GraphQL tested; + class browser tested; + linkStyle 0,1,2,3,4,5,6,7,8,9,10 stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + classDef node color:#909090,fill:#f0f0f0,stroke-width:2px,stroke:#909090 + classDef label stroke-width:0; + classDef tested color:#000000,fill:#a0c0ff,stroke:#6666ff,stroke-width:2px,stroke-dasharray: 5, 5; + + subgraph " " + tested; + mocked; + class tested tested; + end +``` + +#### When to use feature tests + +- Use cases that require a backend and cannot be tested using fixtures. +- Behavior that is not part of a page bundle but defined globally. + +#### Relevant notes + +A `:js` flag is added to the test to make sure the full environment is loaded. + +```ruby +scenario 'successfully', :js do + sign_in(create(:admin)) +end +``` + +The steps of each test are written using capybara methods ([documentation](https://www.rubydoc.info/gems/capybara)). + +Bear in mind <abbr title="XMLHttpRequest">XHR</abbr> calls might require you to use `wait_for_requests` in between steps, like so: + +```ruby +find('.form-control').native.send_keys(:enter) + +wait_for_requests + +expect(page).not_to have_selector('.card') +``` + +## Test helpers + +### Vuex Helper: `testAction` + +We have a helper available to make testing actions easier, as per [official documentation](https://vuex.vuejs.org/guide/testing.html): + +```javascript +testAction( + actions.actionName, // action + { }, // params to be passed to action + state, // state + [ + { type: types.MUTATION}, + { type: types.MUTATION_1, payload: {}}, + ], // mutations committed + [ + { type: 'actionName', payload: {}}, + { type: 'actionName1', payload: {}}, + ] // actions dispatched + done, +); +``` + +Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/javascripts/ide/stores/actions_spec.js). + +### Vue Helper: `mountComponent` + +To make mounting a Vue component easier and more readable, we have a few helpers available in `spec/helpers/vue_mount_component_helper`: + +- `createComponentWithStore` +- `mountComponentWithStore` + +Examples of usage: + +```javascript +beforeEach(() => { + vm = createComponentWithStore(Component, store); + + vm.$store.state.currentBranchId = 'master'; + + vm.$mount(); +}); +``` + +```javascript +beforeEach(() => { + vm = mountComponentWithStore(Component, { + el: '#dummy-element', + store, + props: { badge }, + }); +}); +``` + +Don't forget to clean up: + +```javascript +afterEach(() => { + vm.$destroy(); +}); +``` + +## Testing with older browsers + +Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or Browserstack using the following steps: + +### Browserstack + +[Browserstack](https://www.browserstack.com/) allows you to test more than 1200 mobile devices and browsers. +You can use it directly through the [live app](https://www.browserstack.com/live) or you can install the [chrome extension](https://chrome.google.com/webstore/detail/browserstack/nkihdmlheodkdfojglpcjjmioefjahjb) for easy access. +You can find the credentials on 1Password, under `frontendteam@gitlab.com`. + +### Firefox + +#### macOS + +You can download any older version of Firefox from the releases FTP server, <https://ftp.mozilla.org/pub/firefox/releases/>: + +1. From the website, select a version, in this case `50.0.1`. +1. Go to the mac folder. +1. Select your preferred language, you will find the dmg package inside, download it. +1. Drag and drop the application to any other folder but the `Applications` folder. +1. Rename the application to something like `Firefox_Old`. +1. Move the application to the `Applications` folder. +1. Open up a terminal and run `/Applications/Firefox_Old.app/Contents/MacOS/firefox-bin -profilemanager` to create a new profile specific to that Firefox version. +1. Once the profile has been created, quit the app, and run it again like normal. You now have a working older Firefox version. + --- [Return to Testing documentation](index.md) diff --git a/doc/development/testing_guide/img/qa_on_merge_requests_cicd_architecture.png b/doc/development/testing_guide/img/qa_on_merge_requests_cicd_architecture.png Binary files differdeleted file mode 100644 index 5b93a05db96..00000000000 --- a/doc/development/testing_guide/img/qa_on_merge_requests_cicd_architecture.png +++ /dev/null diff --git a/doc/development/testing_guide/img/review_apps_cicd_architecture.png b/doc/development/testing_guide/img/review_apps_cicd_architecture.png Binary files differdeleted file mode 100644 index 1ee28d3db91..00000000000 --- a/doc/development/testing_guide/img/review_apps_cicd_architecture.png +++ /dev/null diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index aadbea1a540..96e8c30a679 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -22,62 +22,44 @@ automated testing means, and what are its principles: - [Five Factor Testing](https://www.devmynd.com/blog/five-factor-testing): Why do we need tests? - [Principles of Automated Testing](http://www.lihaoyi.com/post/PrinciplesofAutomatedTesting.html): Levels of testing. Prioritize tests. Cost of tests. ---- - ## [Testing levels](testing_levels.md) Learn about the different testing levels, and how to decide at what level your changes should be tested. ---- - ## [Testing best practices](best_practices.md) Everything you should know about how to write good tests: Test Design, RSpec, FactoryBot, system tests, parameterized tests etc. ---- - ## [Frontend testing standards and style guidelines](frontend_testing.md) Everything you should know about how to write good Frontend tests: Karma, testing promises, stubbing etc. ---- - ## [Flaky tests](flaky_tests.md) What are flaky tests, the different kind of flaky tests we encountered, and what we do about them. ---- - ## [GitLab tests in the Continuous Integration (CI) context](ci.md) How GitLab test suite is run in the CI context: setup, caches, artifacts, parallelization, monitoring. ---- - ## [Review apps](review_apps.md) How review apps are set up for GitLab CE/EE and how to use them. ---- - ## [Testing Rake tasks](testing_rake_tasks.md) Everything you should know about how to test Rake tasks. ---- - ## [End-to-end tests](end_to_end/index.md) Everything you should know about how to run end-to-end tests using [GitLab QA][gitlab-qa] testing framework. ---- - [Return to Development documentation](../README.md) [RSpec]: https://github.com/rspec/rspec-rails#feature-specs diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index ae40d628717..11449712a04 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -8,38 +8,33 @@ Review Apps are automatically deployed by each pipeline, both in ### CI/CD architecture diagram - - -<details> -<summary>Show mermaid source</summary> -<pre> +```mermaid graph TD build-qa-image -.->|once the `prepare` stage is done| gitlab:assets:compile review-build-cng -->|triggers a CNG-mirror pipeline and wait for it to be done| CNG-mirror review-build-cng -.->|once the `test` stage is done| review-deploy review-deploy -.->|once the `review` stage is done| review-qa-smoke -subgraph 1. gitlab-ce/ee `prepare` stage +subgraph "1. gitlab-ce/ee `prepare` stage" build-qa-image end -subgraph 2. gitlab-ce/ee `test` stage +subgraph "2. gitlab-ce/ee `test` stage" gitlab:assets:compile -->|plays dependent job once done| review-build-cng end -subgraph 3. gitlab-ce/ee `review` stage - review-deploy["review-deploy<br /><br />Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br /><br />Cloud Native images are deployed to the `review-apps-ce` or `review-apps-ee`<br />Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."] +subgraph "3. gitlab-ce/ee `review` stage" + review-deploy["review-deploy<br><br>Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps-ce` or `review-apps-ee`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."] end -subgraph 4. gitlab-ce/ee `qa` stage - review-qa-smoke[review-qa-smoke<br /><br />gitlab-qa runs the smoke suite against the Review App.] +subgraph "4. gitlab-ce/ee `qa` stage" + review-qa-smoke[review-qa-smoke<br><br>gitlab-qa runs the smoke suite against the Review App.] end -subgraph CNG-mirror pipeline +subgraph "CNG-mirror pipeline" CNG-mirror>Cloud Native images are built]; end -</pre> -</details> +``` ### Detailed explanation @@ -115,6 +110,28 @@ On every [pipeline][gitlab-pipeline] in the `qa` stage, the browser performance testing using a [Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). +## Cluster configuration + +### Node pools + +Both `review-apps-ce` and `review-apps-ee` clusters are currently set up with +two node pools: + +- a node pool of non-preemptible `n1-standard-2` (2 vCPU, 7.5 GB memory) nodes + dedicated to the `tiller` deployment (see below) with a single node. +- a node pool of preemptible `n1-standard-2` (2 vCPU, 7.5 GB memory) nodes, + with a minimum of 1 node and a maximum of 250 nodes. + +### Helm/Tiller + +The `tiller` deployment (the Helm server) is deployed to a dedicated node pool +that has the `app=helm` label and a specific +[taint](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) +to prevent other pods from being scheduled on this node pool. + +This is to ensure Tiller isn't affected by "noisy" neighbors that could put +their node under pressure. + ## How to: ### Log into my Review App @@ -137,8 +154,8 @@ secure note named **gitlab-{ce,ee} Review App's root password**. ### Run a Rails console -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps) - , e.g. `review-qa-raise-e-12chm0`. +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), + e.g. `review-qa-raise-e-12chm0`. 1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. 1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`. 1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. @@ -196,7 +213,7 @@ For the record, the debugging steps to find out this issue were: 1. `kubectl describe pod <pod name>` & confirm exact error message 1. Web search for exact error message, following rabbit hole to [a relevant kubernetes bug report](https://github.com/kubernetes/kubernetes/issues/57345) 1. Access the node over SSH via the GCP console (**Computer Engine > VM - instances** then click the "SSH" button for the node where the `dns-gitlab-review-app-external-dns` pod runs) + instances** then click the "SSH" button for the node where the `dns-gitlab-review-app-external-dns` pod runs) 1. In the node: `systemctl --version` => systemd 232 1. Gather some more information: - `mount | grep kube | wc -l` => e.g. 290 @@ -211,7 +228,7 @@ For the record, the debugging steps to find out this issue were: To resolve the problem, we needed to (forcibly) drain some nodes: 1. Try a normal drain on the node where the `dns-gitlab-review-app-external-dns` - pod runs so that Kubernetes automatically move it to another node: `kubectl drain NODE_NAME` + pod runs so that Kubernetes automatically move it to another node: `kubectl drain NODE_NAME` 1. If that doesn't work, you can also perform a forcible "drain" the node by removing all pods: `kubectl delete pods --field-selector=spec.nodeName=NODE_NAME` 1. In the node: - Perform `systemctl daemon-reload` to remove the dead/inactive units @@ -238,17 +255,8 @@ that a machine will hit the "too many mount points" problem in the future. thousands of unused Docker images.** > We have to start somewhere and improve later. Also, we're using the - CNG-mirror project to store these Docker images so that we can just wipe out - the registry at some point, and use a new fresh, empty one. - -**How big are the Kubernetes clusters (`review-apps-ce` and `review-apps-ee`)?** - - > The clusters are currently set up with a single pool of preemptible nodes, - with a minimum of 1 node and a maximum of 500 nodes. - -**What are the machine running on the cluster?** - - > We're currently using `n1-standard-1` (1 vCPU, 3.75 GB memory) machines. + > CNG-mirror project to store these Docker images so that we can just wipe out + > the registry at some point, and use a new fresh, empty one. **How do we secure this from abuse? Apps are open to the world so we need to find a way to limit it to only us.** diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index e1ce4d3b7d1..0090c84cbf0 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -63,10 +63,9 @@ They're useful to test permissions, redirections, what view is rendered etc. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | -| `app/controllers/` | `spec/controllers/` | RSpec | | +| `app/controllers/` | `spec/controllers/` | RSpec | For N+1 tests, use [request specs](../query_recorder.md#use-request-specs-instead-of-controller-specs) | | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | -| `lib/ci/api/` | `spec/requests/ci/api/` | RSpec | | | `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | ### About controller tests diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 11aafd7b639..7c926c83a36 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -199,7 +199,7 @@ more common ones here. A full list of all the available nodes and their descriptions can be found in the [PostgreSQL source file -"plannodes.h"](https://github.com/postgres/postgres/blob/master/src/include/nodes/plannodes.h) +"plannodes.h"](https://gitlab.com/postgres/postgres/blob/master/src/include/nodes/plannodes.h) ### Seq Scan @@ -224,7 +224,7 @@ used when we would read too much data from an index scan, but too little to perform a sequential scan. A bitmap scan uses what is known as a [bitmap index](https://en.wikipedia.org/wiki/Bitmap_index) to perform its work. -The [source code of PostgreSQL](https://github.com/postgres/postgres/blob/1c2cb2744bf3d8ad751cd5cf3b347f10f48492b3/src/include/nodes/plannodes.h#L446-L457) +The [source code of PostgreSQL](https://gitlab.com/postgres/postgres/blob/REL_11_STABLE/src/include/nodes/plannodes.h#L441) states the following on bitmap scans: > Bitmap Index Scan delivers a bitmap of potential tuple locations; it does not diff --git a/doc/development/uploads.md b/doc/development/uploads.md new file mode 100644 index 00000000000..681ce9d9fe8 --- /dev/null +++ b/doc/development/uploads.md @@ -0,0 +1,270 @@ +# Uploads development documentation + +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. +To prevent occupying a ruby process on I/O operations, we process the upload in workhorse, where is cheaper. +This process can also directly upload to object storage. + +## The problem description + +The following graph explains machine boundaries in a scalable GitLab installation. Without any workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->workhorse + + subgraph "web or API fleet" + workhorse-- 2 -->rails + end + rails-- "3 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +We have three challenges here: performance, availability, and scalability. + +### Performance + +Rails process are expensive in terms of both CPU and memory. Ruby [global interpreter lock](https://en.wikipedia.org/wiki/Global_interpreter_lock) adds to cost too because the ruby process will spend time on I/O operations on step 3 causing incoming requests to pile up. + +In order to improve this, [workhorse disk acceleration](#workhorse-disk-acceleration) was implemented. With this, Rails no longer deals with writing uploaded files to disk. + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->workhorse + + subgraph "web or API fleet" + workhorse-- "3 (without files)" -->rails + end + workhorse -- "2 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +### Availability + +There's also an availability problem in this setup, NFS is a [single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure). + +To address this problem an HA object storage can be used and it's supported by [workhorse object storage acceleration](#workhorse-object-storage-acceleration) + +### Scalability + +Scaling NFS is outside of our support scope, and NFS is not a part of cloud native installations. + +All features that require sidekiq and do not use object storage acceleration won't work without NFS. In Kubernetes, machine boundaries translate to PODs, and in this case the uploaded file will be written into the POD private disk. Since sidekiq POD cannot reach into other pods, the operation will fail to read it. + +## How to select the proper level of acceleration? + +Selecting the proper acceleration is a tradeoff between speed of development and operational costs. + +We can identify three major use-cases for an upload: + +1. **storage:** if we are uploading for storing a file (i.e. artifacts, packages, discussion attachments). In this case [object storage acceleration](#workhorse-object-storage-acceleration) is the proper level as it's the less resource-intensive operation. Additional information can be found on [File Storage in GitLab](file_storage.md). +1. **in-controller/synchronous processing:** if we allow processing **small files** synchronously, using [disk acceleration](#workhorse-disk-acceleration) may speed up development. +1. **sidekiq/asynchronous processing:** Async processing must implement [object storage acceleration](#workhorse-object-storage-acceleration), the reason being that it's the only way to support Cloud Native deployments without a shared NFS. + +For more details about currently broken feature see [epic &1802](https://gitlab.com/groups/gitlab-org/-/epics/1802). + +### Handling repository uploads + +Some features involves git repository uploads without using a regular git client. +Some examples are uploading a repository file from the web interface and [design management](../user/project/issues/design_management.md). + +Those uploads requires the rails controller to act as a git client in lieu of the user. +Those operation falls into _in-controller/synchronous processing_ category, but we have no warranties on the file size. + +In case of a LFS upload, the file pointer is committed synchronously, but file upload to object storage is performed asynchronously with sidekiq. + +## Upload encodings + +By upload encoding we mean how the file is included within the incoming request. + +We have three kinds of file encoding in our uploads: + +1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. +1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. +1. <i class="fa fa-times-circle"></i> **JSON**: some JSON API uploads files as base64 encoded strings. This requires [gitlab-workhorse#226](https://gitlab.com/gitlab-org/gitlab-workhorse/issues/226) to be implemented. + +## Uploading technologies + +By uploading technologies we mean how all the involved services interact with each other. + +GitLab supports 3 kinds of uploading technologies, here follows a brief description with a sequence diagram for each one. Diagrams are not meant to be exhaustive. + +### Regular rails upload + +This is the default kind of upload, and it's most expensive in terms of resources. + +In this case, workhorse is unaware of files being uploaded and acts as a regular proxy. + +When a multipart request reaches the rails application, `Rack::Multipart` leaves behind tempfiles in `/tmp` and uses valuable Ruby process time to copy files around. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + + activate c + c ->>+w: POST /some/url/upload + w->>+r: POST /some/url/upload + + r->>r: save the incoming file on /tmp + r->>r: read the file for processing + + r-->>-c: request result + deactivate c + deactivate w +``` + +### Workhorse disk acceleration + +This kind of upload avoids wasting resources caused by handling upload writes to `/tmp` in rails. + +This optimization is not active by default on REST API requests. + +When enabled, Workhorse looks for files in multipart MIME requests, uploading +any it finds to a temporary file on shared storage. The MIME data in the request +is replaced with the path to the corresponding file before it is forwarded to +Rails. + +To prevent abuse of this feature, Workhorse signs the modified request with a +special header, stating which entries it modified. Rails will ignore any +unsigned path entries. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant s as NFS + + activate c + c ->>+w: POST /some/url/upload + + w->>+s: save the incoming file on a temporary location + s-->>-w: + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+s: read file + s-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +### Workhorse object storage acceleration + +This is the more advanced acceleration technique we have in place. + +Workhorse asks rails for temporary pre-signed object storage URLs and directly uploads to object storage. + +In this setup an extra rails route needs to be implemented in order to handle authorization, +you can see an example of this in [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/controllers/projects/lfs_storage_controller.rb) +and [its routes](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0/config/routes/git_http.rb#L31-32). + +**note:** this will fallback to _Workhorse disk acceleration_ when object storage is not enabled in the gitlab instance. The answer to the `/authorize` call will only contain a file system path. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant os as Object Storage + + activate c + c ->>+w: POST /some/url/upload + + w ->>+r: POST /some/url/upload/authorize + Note over w,r: this request has an empty body + r-->>-w: presigned OS URL + + w->>+os: PUT file + Note over w,os: file is stored on a temporary location. Rails select the destination + os-->>-w: + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + r->>+os: move object to final destination + os-->>-r: + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+os: get object + os-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +## What does the `direct_upload` setting mean? + +[Object storage setting](../administration/uploads.md#object-storage-settings) allows instance administators to enable `direct_upload`, this in an option that only affects the behavior of [workhorse object storage acceleration](#workhorse-object-storage-acceleration). + +This option affect the response to the `/authorize` call. When not enabled, the API response will not contain presigned URLs and workhorse will write the file the shared disk, on the path is provided by rails, acting like object storage was disabled. + +Once the request reachs rails, it will schedule an object storage upload as a sidekiq job. diff --git a/doc/development/utilities.md b/doc/development/utilities.md index 0e396baccff..4021756343c 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -6,44 +6,44 @@ We developed a number of utilities to ease development. - Deep merges an array of hashes: - ``` ruby - Gitlab::Utils::MergeHash.merge( - [{ hello: ["world"] }, - { hello: "Everyone" }, - { hello: { greetings: ['Bonjour', 'Hello', 'Hallo', 'Dzien dobry'] } }, - "Goodbye", "Hallo"] - ) - ``` - - Gives: - - ``` ruby - [ - { - hello: - [ - "world", - "Everyone", - { greetings: ['Bonjour', 'Hello', 'Hallo', 'Dzien dobry'] } - ] - }, - "Goodbye" - ] - ``` + ``` ruby + Gitlab::Utils::MergeHash.merge( + [{ hello: ["world"] }, + { hello: "Everyone" }, + { hello: { greetings: ['Bonjour', 'Hello', 'Hallo', 'Dzien dobry'] } }, + "Goodbye", "Hallo"] + ) + ``` + + Gives: + + ``` ruby + [ + { + hello: + [ + "world", + "Everyone", + { greetings: ['Bonjour', 'Hello', 'Hallo', 'Dzien dobry'] } + ] + }, + "Goodbye" + ] + ``` - Extracts all keys and values from a hash into an array: - ``` ruby - Gitlab::Utils::MergeHash.crush( - { hello: "world", this: { crushes: ["an entire", "hash"] } } - ) - ``` + ``` ruby + Gitlab::Utils::MergeHash.crush( + { hello: "world", this: { crushes: ["an entire", "hash"] } } + ) + ``` - Gives: + Gives: - ``` ruby - [:hello, "world", :this, :crushes, "an entire", "hash"] - ``` + ``` ruby + [:hello, "world", :this, :crushes, "an entire", "hash"] + ``` ## [`Override`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/utils/override.rb) @@ -53,9 +53,9 @@ We developed a number of utilities to ease development. `ENV['STATIC_VERIFICATION']` is set to avoid production runtime overhead. This is useful to check: - - If we have typos in overriding methods. - - If we renamed the overridden methods, making original overriding methods - overrides nothing. + - If we have typos in overriding methods. + - If we renamed the overridden methods, making original overriding methods + overrides nothing. Here's a simple example: @@ -94,47 +94,47 @@ We developed a number of utilities to ease development. - Memoize the value even if it is `nil` or `false`. - We often do `@value ||= compute`, however this doesn't work well if - `compute` might eventually give `nil` and we don't want to compute again. - Instead we could use `defined?` to check if the value is set or not. - However it's tedious to write such pattern, and `StrongMemoize` would - help us use such pattern. + We often do `@value ||= compute`, however this doesn't work well if + `compute` might eventually give `nil` and we don't want to compute again. + Instead we could use `defined?` to check if the value is set or not. + However it's tedious to write such pattern, and `StrongMemoize` would + help us use such pattern. - Instead of writing patterns like this: + Instead of writing patterns like this: - ``` ruby - class Find - def result - return @result if defined?(@result) + ``` ruby + class Find + def result + return @result if defined?(@result) - @result = search - end + @result = search end - ``` + end + ``` - We could write it like: + We could write it like: - ``` ruby - class Find - include Gitlab::Utils::StrongMemoize + ``` ruby + class Find + include Gitlab::Utils::StrongMemoize - def result - strong_memoize(:result) do - search - end + def result + strong_memoize(:result) do + search end end - ``` + end + ``` - Clear memoization - ``` ruby - class Find - include Gitlab::Utils::StrongMemoize - end + ``` ruby + class Find + include Gitlab::Utils::StrongMemoize + end - Find.new.clear_memoization(:result) - ``` + Find.new.clear_memoization(:result) + ``` ## [`RequestCache`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/cache/request_cache.rb) diff --git a/doc/development/ux_guide/animation.md b/doc/development/ux_guide/animation.md index 583ff19bc69..a998ab74a96 100644 --- a/doc/development/ux_guide/animation.md +++ b/doc/development/ux_guide/animation.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://design.gitlab.com/foundations/motion' +redirect_to: 'https://design.gitlab.com/product-foundations/motion' --- -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com). +The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/product-foundations/motion). diff --git a/doc/development/ux_guide/illustrations.md b/doc/development/ux_guide/illustrations.md index ed072b6515f..3592d25c95d 100644 --- a/doc/development/ux_guide/illustrations.md +++ b/doc/development/ux_guide/illustrations.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://design.gitlab.com/foundations/illustration/' +redirect_to: 'https://design.gitlab.com/product-foundations/illustration' --- -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). +The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/product-foundations/illustration). diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index 661ab9cef1a..6b4995aebe2 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -1,15 +1,15 @@ # Verifying Database Capabilities -Sometimes certain bits of code may only work on a certain database and/or +Sometimes certain bits of code may only work on a certain database version. While we try to avoid such code as much as possible sometimes it is necessary to add database (version) specific behaviour. To facilitate this we have the following methods that you can use: -- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used -- `Gitlab::Database.mysql?`: returns `true` if MySQL is being used +- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used. + You can normally just assume this is the case. - `Gitlab::Database.version`: returns the PostgreSQL version number as a string - in the format `X.Y.Z`. This method does not work for MySQL + in the format `X.Y.Z`. This allows you to write code such as: @@ -25,7 +25,7 @@ else end ``` -# Read-only database +## Read-only database The database can be used in read-only mode. In this case we have to make sure all GET requests don't attempt any write operations to the diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 24edd05da2f..944bf5900c5 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -7,9 +7,8 @@ downtime. ## Adding Columns -On PostgreSQL you can safely add a new column to an existing table as long as it -does **not** have a default value. For example, this query would not require -downtime: +You can safely add a new column to an existing table as long as it does **not** +have a default value. For example, this query would not require downtime: ```sql ALTER TABLE projects ADD COLUMN random_value int; @@ -27,11 +26,6 @@ This requires updating every single row in the `projects` table so that indexes in a table. This in turn acquires enough locks on the table for it to effectively block any other queries. -As of MySQL 5.6 adding a column to a table is still quite an expensive -operation, even when using `ALGORITHM=INPLACE` and `LOCK=NONE`. This means -downtime _may_ be required when modifying large tables as otherwise the -operation could potentially take hours to complete. - Adding a column with a default value _can_ be done without requiring downtime when using the migration helper method `Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works @@ -140,7 +134,7 @@ done without requiring downtime. However, this does require that any application changes are deployed _first_. Thus, changing the constraints of a column should happen in a post-deployment migration. NOTE: Avoid using `change_column` as it produces inefficient query because it re-defines -the whole column type. For example, to add a NOT NULL constraint, prefer `change_column_null ` +the whole column type. For example, to add a NOT NULL constraint, prefer `change_column_null` ## Changing Column Types @@ -311,8 +305,7 @@ migrations](background_migrations.md#cleaning-up). ## Adding Indexes Adding indexes is an expensive process that blocks INSERT and UPDATE queries for -the duration. When using PostgreSQL one can work around this by using the -`CONCURRENTLY` option: +the duration. You can work around this by using the `CONCURRENTLY` option: ```sql CREATE INDEX CONCURRENTLY index_name ON projects (column_name); @@ -336,17 +329,9 @@ end Note that `add_concurrent_index` can not be reversed automatically, thus you need to manually define `up` and `down`. -When running this on PostgreSQL the `CONCURRENTLY` option mentioned above is -used. On MySQL this method produces a regular `CREATE INDEX` query. - -MySQL doesn't really have a workaround for this. Supposedly it _can_ create -indexes without the need for downtime but only for variable width columns. The -details on this are a bit sketchy. Since it's better to be safe than sorry one -should assume that adding indexes requires downtime on MySQL. - ## Dropping Indexes -Dropping an index does not require downtime on both PostgreSQL and MySQL. +Dropping an index does not require downtime. ## Adding Tables @@ -370,7 +355,7 @@ transaction this means this approach would require downtime. GitLab allows you to work around this by using `Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method -ensures that when PostgreSQL is used no downtime is needed. +ensures that no downtime is needed. ## Removing Foreign Keys diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index fd16047b8e4..fc3d36910f2 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -1,15 +1,18 @@ --- comments: false +type: index --- # GitLab basics guides -This section provides resources to help you start with GitLab by focusing on basic functionality. +This section provides resources to help you start working with GitLab and Git by focusing +on the basic features that you will need to use. This documentation is split into the following groups: - [GitLab-specific functionality](#gitlab-basics), for basic GitLab features. -- [General Git functionality](#git-basics), for working with Git in conjunction with GitLab. +- [General Git functionality](#working-with-git-from-the-command-line), for working + with Git in conjunction with GitLab. ## GitLab basics @@ -17,22 +20,26 @@ The following are guides to basic GitLab functionality: - [Create and add your SSH public key](create-your-ssh-keys.md), for enabling Git over SSH. - [Create a project](create-project.md), to start using GitLab. -- [Create a group](../user/group/index.md#create-a-new-group), to combine and administer projects together. +- [Create a group](../user/group/index.md#create-a-new-group), to combine and administer + projects together. - [Create a branch](create-branch.md), to make changes to files stored in a project's repository. - [Fork a project](fork-project.md), to duplicate projects so they can be worked on in parallel. - [Add a file](add-file.md), to add new files to a project's repository. -- [Add an image](add-image.md), to add new images to a project's repository. -- [Create an issue](../user/project/issues/managing_issues.md#create-a-new-issue), to start collaborating within a project. -- [Create a merge request](add-merge-request.md), to request changes made in a branch be merged into a project's repository. -- See how these features come together in the [GitLab Flow introduction video](https://youtu.be/InKNIvky2KE) and [GitLab Flow page](../workflow/gitlab_flow.md). +- [Create an issue](../user/project/issues/managing_issues.md#create-a-new-issue), + to start collaborating within a project. +- [Create a merge request](add-merge-request.md), to request changes made in a branch + be merged into a project's repository. +- See how these features come together in the [GitLab Flow introduction video](https://youtu.be/InKNIvky2KE) + and [GitLab Flow page](../workflow/gitlab_flow.md). -## Git basics +## Working with Git from the command line -If you're familiar with Git on the command line, you can interact with your GitLab projects just as you would with any other Git repository. +If you're familiar with Git on the command line, you can interact with your GitLab +projects just as you would with any other Git repository. These resources will help get further acclimated to working on the command line. - [Start using Git on the command line](start-using-git.md), for some simple Git commands. - [Command line basics](command-line-commands.md), to create and edit files using the command line. -More Git resources are available at GitLab's [Git documentation](../topics/git/index.md). +More Git resources are available in GitLab's [Git documentation](../topics/git/index.md). diff --git a/doc/gitlab-basics/add-file.md b/doc/gitlab-basics/add-file.md index e9fbcbc23a9..d547584bf80 100644 --- a/doc/gitlab-basics/add-file.md +++ b/doc/gitlab-basics/add-file.md @@ -1,5 +1,91 @@ -# How to add a file +--- +type: howto +--- -You can create a file in your [terminal](command-line-commands.md) and push -to GitLab or you can use the +# Add a file to a repository + +Adding files to a repository is a small, but key task. Bringing files in to a repository, +such as code, images, or documents, allows them to be tracked by Git, even though they +may have been created elsewhere. + +You can add a file to a repository in your [terminal](#add-a-file-using-the-command-line), and +then push to GitLab. You can also use the [web interface](../user/project/repository/web_editor.md#upload-a-file), +which may be a simpler solution. + +If you need to create a file first, for example a `README.md` text file, that can +also be done from the [terminal](command-line-commands.md#create-a-text-file-in-the-current-directory) or [web interface](../user/project/repository/web_editor.md#create-a-file). + +## Add a file using the command line + +Open a [terminal/shell](command-line-commands.md), and change into the folder of your +GitLab project. This usually means running the following command until you get +to the desired destination: + +```sh +cd <destination folder> +``` + +[Create a branch](create-branch.md) to add your file to, before it is added to the master +(main) branch of the project. It is not strictly necessary, but working directly in +the `master` branch is not recommended unless your project is very small, and you are +the only person working on it. You can [switch to an existing branch](start-using-git.md#work-on-an-existing-branch), +if you have one already. + +Using your standard tool for copying files (for example, Finder in macOS, or File Explorer +in Windows), put the file into a directory within the GitLab project. + +Check if your file is actually present in the directory (if you are in Windows, +use `dir` instead): + +```sh +ls +``` + +You should see the name of the file in the list shown. + +Check the status: + +```sh +git status +``` + +Your file's name should appear in red, so `git` took notice of it! Now add it +to the repository: + +```sh +git add <name of file> +``` + +Check the status again, your file's name should have turned green: + +```sh +git status +``` + +Commit (save) your file to the repository: + +```sh +git commit -m "DESCRIBE COMMIT IN A FEW WORDS" +``` + +Now you can push (send) your changes (in the branch `<branch-name>`) to GitLab +(the Git remote named 'origin'): + +```sh +git push origin <branch-name> +``` + +Your image will be added to your branch in your repository in GitLab. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/gitlab-basics/add-image.md b/doc/gitlab-basics/add-image.md index 17c210c43e0..11a4a6bbd97 100644 --- a/doc/gitlab-basics/add-image.md +++ b/doc/gitlab-basics/add-image.md @@ -1,56 +1,5 @@ -# How to add an image +--- +redirect_to: 'add-file.md' +--- -Using your standard tool for copying files (e.g. Finder in Mac OS, or Explorer -in Windows, or...), put the image file into the GitLab project. You can find the -project as a regular folder in your files. - -Go to your [shell](command-line-commands.md), and move into the folder of your -GitLab project. This usually means running the following command until you get -to the desired destination: - -``` -cd NAME-OF-FOLDER-YOU'D-LIKE-TO-OPEN -``` - -Check if your image is actually present in the directory (if you are in Windows, -use `dir` instead): - -``` -ls -``` - -You should see the name of the image in the list shown. - -Check the status: - -``` -git status -``` - -Your image's name should appear in red, so `git` took notice of it! Now add it -to the repository: - -``` -git add NAME-OF-YOUR-IMAGE -``` - -Check the status again, your image's name should have turned green: - -``` -git status -``` - -Commit: - -``` -git commit -m "DESCRIBE COMMIT IN A FEW WORDS" -``` - -Now you can push (send) your changes (in the branch NAME-OF-BRANCH) to GitLab -(the git remote named 'origin'): - -``` -git push origin NAME-OF-BRANCH -``` - -Your image will be added to your branch in your repository in GitLab. +This document was moved to [another location](add-file.md). diff --git a/doc/gitlab-basics/add-merge-request.md b/doc/gitlab-basics/add-merge-request.md index 7bb2e014738..1a6a26152fa 100644 --- a/doc/gitlab-basics/add-merge-request.md +++ b/doc/gitlab-basics/add-merge-request.md @@ -1,9 +1,16 @@ +--- +type: howto +--- + # How to create a merge request -Merge requests are useful to integrate separate changes that you've made to a -project, on different branches. This is a brief guide on how to create a merge -request. For more information, check the -[merge requests documentation](../user/project/merge_requests/index.md). +Merge requests are how you integrate separate changes that you've made in a +[branch](create-branch.md) to a [project](create-project.md). + +This is a brief guide on how to create a merge request. For more detailed information, +check the [merge requests documentation](../user/project/merge_requests/index.md), or +you can watch our [GitLab Flow video](https://www.youtube.com/watch?v=InKNIvky2KE) for +a quick overview of working with merge requests. --- @@ -12,19 +19,31 @@ request. For more information, check the 1. Go to the project where you'd like to merge your changes and click on the **Merge requests** tab. 1. Click on **New merge request** on the right side of the screen. -1. From there on, you have the option to select the source branch and the target +1. From there, you have the option to select the source branch and the target branch you'd like to compare to. The default target project is the upstream repository, but you can choose to compare across any of its forks. -  +  1. When ready, click on the **Compare branches and continue** button. 1. At a minimum, add a title and a description to your merge request. Optionally, - select a user to review your merge request and to accept or close it. You may - also select a milestone and labels. + select a user to review your merge request. You may also select a milestone and + labels. -  +  1. When ready, click on the **Submit merge request** button. -Your merge request will be ready to be approved and merged. +Your merge request will be ready to be reviewed, approved, and merged. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/gitlab-basics/basic-git-commands.md b/doc/gitlab-basics/basic-git-commands.md index c2a3415cbc4..4e0cc2de2bc 100644 --- a/doc/gitlab-basics/basic-git-commands.md +++ b/doc/gitlab-basics/basic-git-commands.md @@ -1,3 +1,5 @@ -# Basic Git commands +--- +redirect_to: 'start-using-git.md' +--- -This section is now merged into [Start using Git](start-using-git.md). +This document was moved to [another location](start-using-git.md). diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index b7e6844f43a..74539b33642 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -1,75 +1,47 @@ -# Command Line basic commands - -## Start working on your project - -In Git, when you copy a project you say you "clone" it. To work on a git project locally (from your own computer), you will need to clone it. To do this, sign in to GitLab. - -When you are on your Dashboard, click on the project that you'd like to clone. -To work in the project, you can copy a link to the Git repository through a SSH -or a HTTPS protocol. SSH is easier to use after it's been -[set up](create-your-ssh-keys.md). While you are at the **Project** tab, select -HTTPS or SSH from the dropdown menu and copy the link using the _Copy URL to clipboard_ -button (you'll have to paste it on your shell in the next step). - - - -## Working with project files on the command line - -This section has examples of some basic shell commands that you might find useful. For more information, search the web for _bash commands_. - -Alternatively, you can edit files using your choice of editor (IDE) or the GitLab user interface. - -### Clone your project - -Go to your computer's shell and type the following command with your SSH or HTTPS URL: +--- +type: howto, reference +--- -``` -git clone PASTE HTTPS OR SSH HERE -``` - -A clone of the project will be created in your computer. - -NOTE: **Note:** -If you clone your project via a URL that contains special characters, make sure -that characters are URL-encoded. - -### Go into a project directory to work in it - -``` -cd NAME-OF-PROJECT -``` +# Command Line basic commands -### Go back one directory +When [working with Git from the command line](start-using-git.md), you will need to +use more than just the Git commands. There are several basic commands that you should +learn, in order to make full use of the command line. -``` -cd .. -``` +## Start working on your project -### List what’s in the current directory +To work on a Git project locally (from your own computer), with the command line, +first you will need to [clone (copy) it](start-using-git.md#clone-a-repository) to +your computer. -``` -ls -``` +## Working with files on the command line -### List what’s in the current directory that starts with `a` +This section has examples of some basic shell commands that you might find useful. +For more information, search the web for _bash commands_. -``` -ls a* -``` +Alternatively, you can edit files using your choice of editor (IDE), or the GitLab user +interface (not locally). -### List what’s in the current directory that ends with `.md` +### Common commands -``` -ls *.md -``` +The list below is not exhaustive, but contains many of the most commonly used commands. -### Create a new directory +| Command | Description | +|--------------------------------|---------------------------------------------| +| `cd NAME-OF-DIRECTORY` | Go into a directory to work in it | +| `cd ..` | Go back one directory | +| `ls` | List what’s in the current directory | +| `ls a*` | List what’s in the current directory that starts with `a` | +| `ls *.md` | List what’s in the current directory that ends with `.md` | +| `mkdir NAME-OF-YOUR-DIRECTORY` | Create a new directory | +| `cat README.md` | Display the contents of a [text file you created previously](#create-a-text-file-in-the-current-directory) | +| `pwd` | Show the current directory | +| `clear` | Clear the shell window | -``` -mkdir NAME-OF-YOUR-DIRECTORY -``` +### Create a text file in the current directory -### Create a README.md file in the current directory +To create a text file from the command line, for example `README.md`, follow these +steps: ``` touch README.md @@ -80,37 +52,37 @@ nano README.md #### Press: enter ``` -### Show the contents of the README.md file +### Remove a file or directory -``` -cat README.md -``` - -### Remove a file +It is easy to delete (remove) a file or directory, but be careful: DANGER: **Danger:** -This will permanently delete the file. +This will **permanently** delete a file. ``` rm NAME-OF-FILE ``` -### Remove a directory and all of its contents - DANGER: **Danger:** -This will permanently delete the directory and all of its contents. +This will **permanently** delete a directory and **all** of its contents. ``` rm -r NAME-OF-DIRECTORY ``` -### View command history +### View and Execute commands from history + +You can view the history of all the commands you executed from the command line, +and then execute any of them again, if needed. + +First, list the commands you executed previously: ``` history ``` -### Execute command 123 from history +Then, choose a command from the list and check the number next to the command (`123`, +for example) . Execute the same full command with: ``` !123 @@ -118,27 +90,32 @@ history ### Carry out commands for which the account you are using lacks authority -You will be asked for an administrator’s password. +Not all commands can be executed from a basic user account on a computer, you may +need administrator's rights to execute commands that affect the system, or try to access +protected data, for example. You can use `sudo` to execute these commands, but you +will likely be asked for an administrator password. ``` -sudo COMMAND +sudo RESTRICTED-COMMAND ``` CAUTION: **Caution:** Be careful of the commands you run with `sudo`. Certain commands may cause -damage to your data and system. +damage to your data or system. -### Show which directory I am in +## Sample Git taskflow -``` -pwd -``` +If you are completely new to Git, looking through some [sample taskflows](https://rogerdudler.github.io/git-guide/) +will help you understand the best practices for using these commands as you work. -### Clear the shell window +<!-- ## Troubleshooting -``` -clear -``` -### Sample Git taskflow +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -If you are completely new to Git, looking through some [sample taskflows](https://rogerdudler.github.io/git-guide/) will help you understand best practices for using these commands as you work. +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/gitlab-basics/create-branch.md b/doc/gitlab-basics/create-branch.md index ad94f0dad29..e2a2fb52af8 100644 --- a/doc/gitlab-basics/create-branch.md +++ b/doc/gitlab-basics/create-branch.md @@ -1,12 +1,16 @@ +--- +type: howto +--- + # How to create a branch -A branch is an independent line of development. +A branch is an independent line of development in a [project](../user/project/index.md). -New commits are recorded in the history for the current branch, which results -in taking the source from someone’s repository (the place where the history of -your work is stored) at certain point in time, and apply your own changes to it -in the history of the project. +When you create a new branch (in your [terminal](basic-git-commands.md) or with +[the web interface](../user/project/repository/web_editor.md#create-a-new-branch)), +you are creating a snapshot of a certain branch, usually the main `master` branch, +at it's current state. From there, you can start to make your own changes without +affecting the main codebase. The history of your changes will be tracked in your branch. -To add changes to your GitLab project, you should create a branch. You can do -it in your [terminal](basic-git-commands.md) or by -[using the web interface](../user/project/repository/web_editor.md#create-a-new-branch). +When your changes are ready, you then merge them into the rest of the codebase with a +[merge request](add-merge-request.md). diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index ccba72f0ef8..8bbaf5d1927 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -1,10 +1,13 @@ -# Create a project +--- +type: howto +--- -[Projects](../user/project/index.md) combine many features of GitLab together. +# Creating projects -NOTE: **Note:** -For a list of words that cannot be used as project names see -[Reserved project and group names](../user/reserved_names.md). +Most work in GitLab is done within a [Project](../user/project/index.md). Files and +code are saved in projects, and most features are used within the scope of projects. + +## Create a project in GitLab To create a project in GitLab: @@ -14,94 +17,98 @@ To create a project in GitLab: - Create a [blank project](#blank-projects). - Create a project using with one of the available [project templates](#project-templates). - [Import a project](../user/project/import/index.md) from a different repository, - if enabled on your GitLab instance. Contact your GitLab admin if this - is unavailable. + if enabled on your GitLab instance. Contact your GitLab admin if this is unavailable. - Run [CI/CD pipelines for external repositories](../ci/ci_cd_for_external_repos/index.md). **(PREMIUM)** -## Blank projects +NOTE: **Note:** +For a list of words that cannot be used as project names see +[Reserved project and group names](../user/reserved_names.md). + +### Blank projects To create a new blank project on the **New project** page: 1. On the **Blank project** tab, provide the following information: - - The name of your project in the **Project name** field. You can't use - special characters, but you can use spaces, hyphens, underscores or even - emoji. - - The **Project description (optional)** field enables you to enter a - description for your project's dashboard, which will help others - understand what your project is about. Though it's not required, it's a good - idea to fill this in. - - Changing the **Visibility Level** modifies the project's - [viewing and access rights](../public_access/public_access.md) for users. - - Selecting the **Initialize repository with a README** option creates a - README file so that the Git repository is initialized, has a default branch, and - can be cloned. + - The name of your project in the **Project name** field. You can't use + special characters, but you can use spaces, hyphens, underscores or even + emoji. + - The **Project description (optional)** field enables you to enter a + description for your project's dashboard, which will help others + understand what your project is about. Though it's not required, it's a good + idea to fill this in. + - Changing the **Visibility Level** modifies the project's + [viewing and access rights](../public_access/public_access.md) for users. + - Selecting the **Initialize repository with a README** option creates a + README file so that the Git repository is initialized, has a default branch, and + can be cloned. 1. Click **Create project**. -## Project templates +### Project templates -Project templates can pre-populate your project with necessary files to get you started quickly. +Project templates can pre-populate a new project with the necessary files to get you +started quickly. There are two types of project templates: - [Built-in templates](#built-in-templates), sourced from the following groups: - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) - [`pages`](https://gitlab.com/pages) -- [Custom project templates](#custom-project-templates-premium-only), for custom templates configured by GitLab administrators and users. +- [Custom project templates](#custom-project-templates-premium), for custom templates + configured by GitLab administrators and users. -### Built-in templates +#### Built-in templates Built-in templates are project templates that are: -- Developed and maintained in the - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. +- Developed and maintained in the [`project-templates`](https://gitlab.com/gitlab-org/project-templates) + and [`pages`](https://gitlab.com/pages) groups. - Released with GitLab. To use a built-in template on the **New project** page: 1. On the **Create from template** tab, select the **Built-in** tab. 1. From the list of available built-in templates, click the: - - **Preview** button to look at the template source itself. - - **Use template** button to start creating the project. -1. Finish creating the project by filling out the project's details. The process is the same as for - using a [blank project](#blank-projects). + - **Preview** button to look at the template source itself. + - **Use template** button to start creating the project. +1. Finish creating the project by filling out the project's details. The process is + the same as creating a [blank project](#blank-projects). TIP: **Tip:** -You can improve the existing built-in templates or contribute new ones on the -[`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. +You can improve the existing built-in templates or contribute new ones in the +[`project-templates`](https://gitlab.com/gitlab-org/project-templates) and +[`pages`](https://gitlab.com/pages) groups. -### Custom project templates **(PREMIUM ONLY)** +#### Custom project templates **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing) 11.2. -Creating new projects based on custom project templates is a convenient option to bootstrap a project. +Creating new projects based on custom project templates is a convenient option to +bootstrap a project. -Custom projects are available from the **Instance** or **Group** tabs under the **Create from template** tab, -depending on the type of template. +Custom projects are available at the [instance-level](../user/admin_area/custom_project_templates.md) +from the **Instance** tab, or at the [group-level](../user/group/custom_project_templates.md) +from the **Group** tab, under the **Create from template** tab. To use a custom project template on the **New project** page: 1. On the **Create from template** tab, select the **Instance** tab or the **Group** tab. 1. From the list of available custom templates, click the: - - **Preview** button to look at the template source itself. - - **Use template** button to start creating the project. -1. Finish creating the project by filling out the project's details. The process is the same as for - using a [blank project](#blank-projects). - -For information on configuring custom project templates, see: - -- [Custom instance-level project templates](../user/admin_area/custom_project_templates.md), for instance-level templates. -- [Custom group-level project templates](../user/group/custom_project_templates.md), for group-level templates. + - **Preview** button to look at the template source itself. + - **Use template** button to start creating the project. +1. Finish creating the project by filling out the project's details. The process is + the same as creating a [blank project](#blank-projects). ## Push to create a new project > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/26388) in GitLab 10.5. -When you create a new repo locally, instead of going to GitLab to manually -create a new project and then push the repo, you can directly push it to -GitLab to create the new project, all without leaving your terminal. If you have access to that -namespace, we will automatically create a new project under that GitLab namespace with its -visibility set to Private by default (you can later change it in the [project's settings](../public_access/public_access.md#how-to-change-project-visibility)). +When you create a new repository locally, instead of going to GitLab to manually +create a new project and then [clone the repo](start-using-git.md#clone-a-repository) +locally, you can directly push it to GitLab to create the new project, all without leaving +your terminal. If you have access rights to the associated namespace, GitLab will +automatically create a new project under that GitLab namespace with its visibility +set to Private by default (you can later change it in the [project's settings](../public_access/public_access.md#how-to-change-project-visibility)). This can be done by using either SSH or HTTPS: @@ -127,3 +134,15 @@ remote: To view the project, visit: remote: https://gitlab.example.com/namespace/nonexistent-project remote: ``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/gitlab-basics/create-your-ssh-keys.md b/doc/gitlab-basics/create-your-ssh-keys.md index aac73d4c9c5..338b96374aa 100644 --- a/doc/gitlab-basics/create-your-ssh-keys.md +++ b/doc/gitlab-basics/create-your-ssh-keys.md @@ -1,22 +1,27 @@ -# Create and add your SSH public key +--- +type: howto +--- -This topic describes how to: +# Create and add your SSH public key -- Create an SSH key pair to use with GitLab. -- Add the SSH public key to your GitLab account. +It is best practice to use [Git over SSH instead of Git over HTTP](https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols). +In order to use SSH, you will need to -You do this to use [Git over SSH instead of Git over HTTP](https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols). +1. [Create an SSH key pair](#creating-your-ssh-key-pair) on your local computer. +1. [Add the key to GitLab](#adding-your-ssh-public-key-to-gitlab). ## Creating your SSH key pair -1. Go to your [command line](start-using-git.md). -1. Follow the [instructions](../ssh/README.md#generating-a-new-ssh-key-pair) to generate your SSH key pair. +1. Go to your [command line](start-using-git.md#open-a-shell). +1. Follow the [instructions](../ssh/README.md#generating-a-new-ssh-key-pair) to generate + your SSH key pair. ## Adding your SSH public key to GitLab -To add the SSH public key to GitLab, -see [Adding an SSH key to your GitLab account](../ssh/README.md#adding-an-ssh-key-to-your-gitlab-account). +To add the SSH public key to GitLab, see +[Adding an SSH key to your GitLab account](../ssh/README.md#adding-an-ssh-key-to-your-gitlab-account). NOTE: **Note:** -Once you add a key, you cannot edit it. If the paste -[didn't work](../ssh/README.md#testing-that-everything-is-set-up-correctly), you need to remove the key and re-add it. +Once you add a key, you cannot edit it. If it didn't paste properly, it +[will not work](../ssh/README.md#testing-that-everything-is-set-up-correctly), and +you will need to remove the key from GitLab and try adding it again. diff --git a/doc/gitlab-basics/fork-project.md b/doc/gitlab-basics/fork-project.md index a128a7c7dd3..5c19985121d 100644 --- a/doc/gitlab-basics/fork-project.md +++ b/doc/gitlab-basics/fork-project.md @@ -1,20 +1,11 @@ +--- +type: howto +--- + # How to fork a project A fork is a copy of an original repository that you put in another namespace where you can experiment and apply changes that you can later decide whether or not to share, without affecting the original project. -It takes just a few steps to fork a project in GitLab. - -1. Go to a project's dashboard under the **Project** tab and click on the - **Fork** button. - -  - -1. You will be asked where to fork the repository. Click on the user or group - to where you'd like to add the forked project. - -  - -1. After a few moments, depending on the repository's size, the forking will - complete. +It takes just a few steps to [fork a project in GitLab](../workflow/forking_workflow.md#creating-a-fork). diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index b3c5d32f2f5..a289b90b81b 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -1,38 +1,43 @@ +--- +type: howto, tutorial +--- + # Start using Git on the command line -If you want to start using Git and GitLab, make sure that you have created and/or signed into an account on GitLab. +While GitLab has a powerful user interface, if you want to use Git itself, you will +have to do so from the command line. If you want to start using Git and GitLab together, +make sure that you have created and/or signed into an account on GitLab. ## Open a shell -Depending on your operating system, you will need to use a shell of your preference. Here are some suggestions: - -- [Terminal](http://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line) on Mac OSX +Depending on your operating system, you will need to use a shell of your preference. +Here are some suggestions: +- [Terminal](http://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line) on macOS - [GitBash](https://msysgit.github.io) on Windows - - [Linux Terminal](http://www.howtogeek.com/140679/beginner-geek-how-to-start-using-the-linux-terminal/) on Linux ## Check if Git has already been installed -Git is usually preinstalled on Mac and Linux. - -Type the following command and then press enter: +Git is usually preinstalled on Mac and Linux, so run the following command: ```bash git --version ``` -You should receive a message that tells you which Git version you have on your computer. If you don’t receive a "Git version" message, it means that you need to [download Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). +You should receive a message that tells you which Git version you have on your computer. +If you don’t receive a "Git version" message, it means that you need to +[download Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git). -If Git doesn't automatically download, there's an option on the website to [download manually](https://git-scm.com/downloads). Then follow the steps on the installation window. - -After you are finished installing Git, open a new shell and type `git --version` again to verify that it was correctly installed. +After you are finished installing Git, open a new shell and type `git --version` again +to verify that it was correctly installed. ## Add your Git username and set your email -It is important to configure your Git username and email address, since every Git commit will use this information to identify you as the author. +It is important to configure your Git username and email address, since every Git +commit will use this information to identify you as the author. -On your shell, type the following command to add your username: +In your shell, type the following command to add your username: ```bash git config --global user.name "YOUR_USERNAME" @@ -56,7 +61,10 @@ To verify that you entered your email correctly, type: git config --global user.email ``` -You'll need to do this only once, since you are using the `--global` option. It tells Git to always use this information for anything you do on that system. If you want to override this with a different username or email address for specific projects, you can run the command without the `--global` option when you’re in that project. +You'll need to do this only once, since you are using the `--global` option. It tells +Git to always use this information for anything you do on that system. If you want +to override this with a different username or email address for specific projects or repositories, +you can run the command without the `--global` option when you’re in that project, and that will default to `--local`. You can read more on how Git manages configurations in the [Git Config](https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration) documentation. ## Check your information @@ -68,12 +76,12 @@ git config --global --list ## Basic Git commands -Start using Git via the command line with the most basic -commands as described below. +Start using Git via the command line with the most basic commands as described below. -## Initialize a local directory for Git version control +### Initialize a local directory for Git version control -If you have an existing local directory that you want to *initialize* for version control, use the `init` command to instruct Git to begin tracking the directory: +If you have an existing local directory that you want to *initialize* for version +control, use the `init` command to instruct Git to begin tracking the directory: ```bash git init @@ -81,34 +89,32 @@ git init This creates a `.git` directory that contains the Git configuration files. -Once the directory has been initialized, you can [add a remote repository](#add-a-remote-repository) and [send changes to GitLab.com](#send-changes-to-gitlabcom). View the instructions on [Create a project](../gitlab-basics/create-project.html#push-to-create-a-new-project) to create a new project on GitLab with your changes. +Once the directory has been initialized, you can [add a remote repository](#add-a-remote-repository) +and [send changes to GitLab.com](#send-changes-to-gitlabcom). You will also need to +[create a new project in GitLab](../gitlab-basics/create-project.html#push-to-create-a-new-project) +for your Git repository. ### Clone a repository -To start working locally on an existing remote repository, -clone it with the command `git clone <repository path>`. -By cloning a repository, you'll download a copy of its -files into your local computer, preserving the Git -connection with the remote repository. +To start working locally on an existing remote repository, clone it with the command +`git clone <repository path>`. By cloning a repository, you'll download a copy of its +files to your local computer, automatically preserving the Git connection with the +remote repository. -You can either clone it via HTTPS or [SSH](../ssh/README.md). -If you chose to clone it via HTTPS, you'll have to enter your -credentials every time you pull and push. With SSH, you enter -your credentials once and can pull and push straightaway. +You can either clone it via HTTPS or [SSH](../ssh/README.md). If you chose to clone +it via HTTPS, you'll have to enter your credentials every time you pull and push. You can read more about credential storage in the [Git Credentials documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). With SSH, you enter your credentials only once. -You can find both paths (HTTPS and SSH) by navigating to -your project's landing page and clicking **Clone**. GitLab -will prompt you with both paths, from which you can copy +You can find both paths (HTTPS and SSH) by navigating to your project's landing page +and clicking **Clone**. GitLab will prompt you with both paths, from which you can copy and paste in your command line. -As an example, consider a repository path: +As an example, consider this repository path: - HTTPS: `https://gitlab.com/gitlab-org/gitlab-ce.git` -- SSH: `` git@gitlab.com:gitlab-org/gitlab-ce.git `` +- SSH: `git@gitlab.com:gitlab-org/gitlab-ce.git` -To get started, open a terminal window in the directory -you wish to clone the repository files into, and run one -of the following commands. +To get started, open a terminal window in the directory you wish to clone the repository +files into, and run one of the following commands. Clone via HTTPS: @@ -122,13 +128,15 @@ Clone via SSH: git clone git@gitlab.com:gitlab-org/gitlab-ce.git ``` -Both commands will download a copy of the files in a -folder named after the project's name. - -You can then navigate to the directory and start working +Both commands will download a copy of the files in a folder named after the project's +name. You can then navigate to the directory and start working on it locally. -### Go to the master branch to pull the latest changes from there +### Switch to the master branch + +You are always in a branch when working with Git. The main branch is the master branch, +but you can use the same command to switch to a different branch by changing `master` +to the branch name. ```bash git checkout master @@ -136,13 +144,22 @@ git checkout master ### Download the latest changes in the project -This is for you to work on an up-to-date copy (it is important to do this every time you start working on a project), while you set up tracking branches. You pull from remote repositories to get all the changes made by users since the last time you cloned or pulled the project. Later, you can push your local commits to the remote repositories. +To work on an up-to-date copy of the project (it is important to do this every time +you start working on a project), you `pull` to get all the changes made by users since +the last time you cloned or pulled the project. Use `master` for the `<name-of-branch>` +to get the main branch code, or the branch name of the branch you are currently working +in. ```bash -git pull REMOTE NAME-OF-BRANCH +git pull <REMOTE> <name-of-branch> ``` -When you first clone a repository, REMOTE is typically "origin". This is where the repository came from, and it indicates the SSH or HTTPS URL of the repository on the remote server. NAME-OF-BRANCH is usually "master", but it may be any existing branch. +When you clone a repository, `REMOTE` is typically `origin`. This is where the +repository was cloned from, and it indicates the SSH or HTTPS URL of the repository +on the remote server. `<name-of-branch>` is usually `master`, but it may be any existing +branch. You can create additional named remotes and branches as necessary. + +You can learn more on how Git manages remote repositories in the [Git Remote documentation](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes). ### View your remote repositories @@ -152,22 +169,27 @@ To view your remote repositories, type: git remote -v ``` +The `-v` flag stands for verbose. + ### Add a remote repository To add a link to a remote repository: ```bash -git remote add SOURCE-NAME REPOSITORY-PATH +git remote add <source-name> <repository-path> ``` -You'll use this source name every time you [push changes to GitLab.com](#send-changes-to-gitlabcom), so use something easy to remember and type. +You'll use this source name every time you [push changes to GitLab.com](#send-changes-to-gitlabcom), +so use something easy to remember and type. ### Create a branch -To create a branch, type the following (spaces won't be recognized in the branch name, so you will need to use a hyphen or underscore): +To create a new branch, to work from without affecting the `master` branch, type the +following (spaces won't be recognized in the branch name, so you will need to use a +hyphen or underscore): ```bash -git checkout -b NAME-OF-BRANCH +git checkout -b <name-of-branch> ``` ### Work on an existing branch @@ -175,12 +197,14 @@ git checkout -b NAME-OF-BRANCH To switch to an existing branch, so you can work on it: ```bash -git checkout NAME-OF-BRANCH +git checkout <name-of-branch> ``` ### View the changes you've made -It's important to be aware of what's happening and the status of your changes. When you add, change, or delete files/folders, Git knows about it. To check the status of your changes: +It's important to be aware of what's happening and the status of your changes. When +you add, change, or delete files/folders, Git knows about it. To check the status of +your changes: ```bash git status @@ -188,7 +212,8 @@ git status ### View differences -To view the differences between your local, unstaged changes and the repository versions that you cloned or pulled, type: +To view the differences between your local, unstaged changes and the repository versions +that you cloned or pulled, type: ```bash git diff @@ -196,16 +221,19 @@ git diff ### Add and commit local changes -You'll see your local changes in red when you type `git status`. These changes may be new, modified, or deleted files/folders. Use `git add` to stage a local file/folder for committing. Then use `git commit` to commit the staged files: +You'll see any local changes in red when you type `git status`. These changes may +be new, modified, or deleted files/folders. Use `git add` to first stage (prepare) +a local file/folder for committing. Then use `git commit` to commit (save) the staged +files: ```bash -git add FILE OR FOLDER +git add <file-name OR folder-name> git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" ``` ### Add all changes to commit -To add and commit all local changes in one command: +To add and commit (save) all local changes quickly: ```bash git add . @@ -213,39 +241,36 @@ git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" ``` NOTE: **Note:** -The `.` character typically means _all_ in Git. +The `.` character means _all file changes in the current directory and all subdirectories_. ### Send changes to GitLab.com -To push all local commits to the remote repository: +To push all local commits (saved changes) to the remote repository: ```bash -git push REMOTE NAME-OF-BRANCH +git push <remote> <name-of-branch> ``` -For example, to push your local commits to the _master_ branch of the _origin_ remote: +For example, to push your local commits to the _`master`_ branch of the _`origin`_ remote: ```bash git push origin master ``` -### Delete all changes in the Git repository +### Delete all changes in the branch -To delete all local changes in the repository that have not been added to the staging area, and leave unstaged files/folders, type: +To delete all local changes in the branch that have not been added to the staging +area, and leave unstaged files/folders, type: ```bash git checkout . ``` -### Delete all untracked changes in the Git repository - -```bash -git clean -f -``` +Note that this removes *changes* to files, not the files themselves. ### Unstage all changes that have been added to the staging area -To undo the most recent add, but not committed, files/folders: +To undo the most recently added, but not committed, changes to files/folders: ```bash git reset . @@ -259,25 +284,31 @@ To undo the most recent commit, type: git reset HEAD~1 ``` -This leaves the files and folders unstaged in your local repository. +This leaves the changed files and folders unstaged in your local repository. CAUTION: **Warning:** -A Git commit is mostly irreversible, particularly if you already pushed it to the remote repository. Although you can undo a commit, the best option is to avoid the situation altogether. +A Git commit should not usually be reverse, particularly if you already pushed it +to the remote repository. Although you can undo a commit, the best option is to avoid +the situation altogether by working carefully. -### Merge created branch with master branch +### Merge a branch with master branch -You need to be in the created branch. +When you are ready to make all the changes in a branch a permanent addition to +the master branch, you `merge` the two together: ```bash -git checkout NAME-OF-BRANCH +git checkout <name-of-branch> git merge master ``` -### Merge master branch with created branch +<!-- ## Troubleshooting -You need to be in the master branch. +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -```bash -git checkout master -git merge NAME-OF-BRANCH -``` +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/README.md b/doc/install/README.md index af98791c8e9..fd91527ed4c 100644 --- a/doc/install/README.md +++ b/doc/install/README.md @@ -13,7 +13,7 @@ and cost of hosting. There are many ways you can install GitLab depending on your platform: -1. **Omnibus Gitlab**: The official deb/rpm packages that contain a bundle of GitLab +1. **Omnibus GitLab**: The official deb/rpm packages that contain a bundle of GitLab and the various components it depends on like PostgreSQL, Redis, Sidekiq, etc. 1. **GitLab Helm chart**: The cloud native Helm chart for installing GitLab and all its components on Kubernetes. diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 73eaf758923..358ba971049 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -59,10 +59,11 @@ Here's a list of the AWS services we will use, with links to pricing information - **ElastiCache**: An in-memory cache environment will be used to provide a High Availability Redis configuration. See the [Amazon ElastiCache pricing](https://aws.amazon.com/elasticache/pricing/). - + NOTE: **Note:** Please note that while we will be using EBS for storage, we do not recommend using EFS as it may negatively impact GitLab's performance. You can review the [relevant documentation](../../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. ## Creating an IAM EC2 instance role and profile + To minimize the permissions of the user, we'll create a new [IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) role with limited access: @@ -90,7 +91,7 @@ We'll now create a VPC, a virtual networking environment that you'll control: `10.0.0.0/16`. If you don't require dedicated hardware, you can leave "Tenancy" as default. Click **Yes, Create** when ready. -  +  ### Subnets @@ -107,16 +108,16 @@ RDS instances as well: for example `gitlab-public-10.0.0.0`, select the VPC we created previously, and at the IPv4 CIDR block let's give it a 24 subnet `10.0.0.0/24`: -  +  1. Follow the same steps to create all subnets: - | Name tag | Type |Availability Zone | CIDR block | - | -------- | ---- | ---------------- | ---------- | - | gitlab-public-10.0.0.0 | public | us-west-2a | 10.0.0.0 | - | gitlab-private-10.0.1.0 | private | us-west-2a | 10.0.1.0 | - | gitlab-public-10.0.2.0 | public | us-west-2b | 10.0.2.0 | - | gitlab-private-10.0.3.0 | private | us-west-2b | 10.0.3.0 | + | Name tag | Type |Availability Zone | CIDR block | + | -------- | ---- | ---------------- | ---------- | + | gitlab-public-10.0.0.0 | public | us-west-2a | 10.0.0.0 | + | gitlab-private-10.0.1.0 | private | us-west-2a | 10.0.1.0 | + | gitlab-public-10.0.2.0 | public | us-west-2b | 10.0.2.0 | + | gitlab-private-10.0.3.0 | private | us-west-2b | 10.0.3.0 | ### Route Table @@ -139,7 +140,7 @@ create a new one: 1. Select it from the table, and then under the **Actions** dropdown choose "Attach to VPC". -  +  1. Choose `gitlab-vpc` from the list and hit **Attach**. @@ -154,14 +155,14 @@ it receive traffic from any destination. as destination. In the target, select the `gitlab-gateway` we created previously. Hit **Save** once done. -  +  Next, we must associate the **public** subnets to the route table: 1. Select the **Subnet Associations** tab and hit **Edit**. 1. Check only the public subnet and hit **Save**. -  +  --- @@ -178,12 +179,12 @@ The security group is basically the firewall: Inbound Rules tab. You will need to open the SSH, HTTP, and HTTPS ports. Set the source to `0.0.0.0/0`. -  +  - TIP: **Tip:** - Based on best practices, you should allow SSH traffic from only a known - host or CIDR block. In that case, change the SSH source to be custom and give - it the IP you want to SSH from. + TIP: **Tip:** + Based on best practices, you should allow SSH traffic from only a known + host or CIDR block. In that case, change the SSH source to be custom and give + it the IP you want to SSH from. 1. When done, click **Save**. @@ -204,7 +205,7 @@ create the actual RDS instance. we defined them in the [subnets section](#subnets)). Click **Create** when ready. -  +  ### Creating the database @@ -214,27 +215,27 @@ Now, it's time to create the database: 1. Select PostgreSQL and click **Next**. 1. Since this is a production server, let's choose "Production". Click **Next**. 1. Let's see the instance specifications: - 1. Leave the license model as is (`postgresql-license`). - 1. For the version, select the latest of the 9.6 series (check the - [database requirements](../../install/requirements.md#postgresql-requirements)) - if there are any updates on this). - 1. For the size, let's select a `t2.medium` instance. - 1. Multi-AZ-deployment is recommended as redundancy, so choose "Create - replica in different zone". Read more at - [High Availability (Multi-AZ)](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html). - 1. A Provisioned IOPS (SSD) storage type is best suited for HA (though you can - choose a General Purpose (SSD) to reduce the costs). Read more about it at - [Storage for Amazon RDS](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). - -1. The rest of the settings on this page request a DB isntance identifier, username + 1. Leave the license model as is (`postgresql-license`). + 1. For the version, select the latest of the 9.6 series (check the + [database requirements](../../install/requirements.md#postgresql-requirements)) + if there are any updates on this). + 1. For the size, let's select a `t2.medium` instance. + 1. Multi-AZ-deployment is recommended as redundancy, so choose "Create + replica in different zone". Read more at + [High Availability (Multi-AZ)](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html). + 1. A Provisioned IOPS (SSD) storage type is best suited for HA (though you can + choose a General Purpose (SSD) to reduce the costs). Read more about it at + [Storage for Amazon RDS](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). + +1. The rest of the settings on this page request a DB instance identifier, username and a master password. We've chosen to use `gitlab-db-ha`, `gitlab` and a very secure password respectively. Keep these in hand for later. 1. Click **Next** to proceed to the advanced settings. 1. Make sure to choose our gitlab VPC, our subnet group, set public accessibility to **No**, and to leave it to create a new security group. The only additional - change which will be helpful is the database name for which we can use - `gitlabhq_production`. At the very bottom, there's an option to enable - auto updates to minor versions. You may want to turn it off. + change which will be helpful is the database name for which we can use + `gitlabhq_production`. At the very bottom, there's an option to enable + auto updates to minor versions. You may want to turn it off. 1. When done, click **Create database**. ### Installing the `pg_trgm` extension for PostgreSQL @@ -276,7 +277,7 @@ To set up Redis: Make sure to select our VPC and its [private subnets](#subnets). Click **Create** when ready. -  +  1. Select **Redis** on the left menu and click **Create** to create a new Redis cluster. Depending on your load, you can choose whether to enable @@ -284,16 +285,16 @@ To set up Redis: chance to deploy Redis in multi availability zones. In this guide, we chose not to enable it. 1. In the settings section: - 1. Give the cluster a name (`gitlab-redis`) and a description. - 1. For the version, select the latest of `3.2` series (e.g., `3.2.10`). - 1. Select the node type and the number of replicas. + 1. Give the cluster a name (`gitlab-redis`) and a description. + 1. For the version, select the latest of `3.2` series (e.g., `3.2.10`). + 1. Select the node type and the number of replicas. 1. In the advanced settings section: 1. Select the multi-AZ auto-failover option. 1. Select the subnet group we created previously. 1. Manually select the preferred availability zones, and under "Replica 2" choose a different zone than the other two. -  +  1. In the security settings, edit the security groups and choose the `gitlab-security-group` we had previously created. @@ -316,11 +317,11 @@ and add a custom TCP rule for port `6379` accessible within itself. On the EC2 dashboard, look for Load Balancer on the left column: 1. Click the **Create Load Balancer** button. - 1. Choose the Application Load Balancer. - 1. Give it a name (`gitlab-loadbalancer`) and set the scheme to "internet-facing". - 1. In the "Listeners" section, make sure it has HTTP and HTTPS. - 1. In the "Availability Zones" section, select the `gitlab-vpc` we have created - and associate the **public subnets**. + 1. Choose the Application Load Balancer. + 1. Give it a name (`gitlab-loadbalancer`) and set the scheme to "internet-facing". + 1. In the "Listeners" section, make sure it has HTTP and HTTPS. + 1. In the "Availability Zones" section, select the `gitlab-vpc` we have created + and associate the **public subnets**. 1. Click **Configure Security Settings** to go to the next section to select the TLS certificate. When done, go to the next step. 1. In the "Security Groups" section, create a new one by giving it a name @@ -355,7 +356,7 @@ Choose the AMI: where `<version>` the latest version as seen on the [releases page](https://about.gitlab.com/releases/). -  +  ### Choose an instance type @@ -504,19 +505,19 @@ The EBS volume will host the Git repositories data: 1. Tell GitLab to store its data in the new directory by editing `/etc/gitlab/gitlab.rb` with your editor: - ```ruby - git_data_dirs({ - "default" => { "path" => "/mnt/gitlab-data" } - }) - ``` + ```ruby + git_data_dirs({ + "default" => { "path" => "/mnt/gitlab-data" } + }) + ``` - where `/mnt/gitlab-data` the location where you will store the Git data. + where `/mnt/gitlab-data` the location where you will store the Git data. 1. Save the file and reconfigure GitLab: - ```sh - sudo gitlab-ctl reconfigure - ``` + ```sh + sudo gitlab-ctl reconfigure + ``` TIP: **Tip:** If you wish to add more than one data volumes to store the Git repositories, @@ -549,15 +550,15 @@ After you SSH into the instance, configure the domain name: 1. Open `/etc/gitlab/gitlab.rb` with your preferred editor. 1. Edit the `external_url` value: - ```ruby - external_url 'http://example.com' - ``` + ```ruby + external_url 'http://example.com' + ``` 1. Reconfigure GitLab: - ```sh - sudo gitlab-ctl reconfigure - ``` + ```sh + sudo gitlab-ctl reconfigure + ``` You should now be able to reach GitLab at the URL you defined. To use HTTPS (recommended), see the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). @@ -608,9 +609,9 @@ To back up GitLab: 1. SSH into your instance. 1. Take a backup: - ```sh - sudo gitlab-rake gitlab:backup:create - ``` + ```sh + sudo gitlab-backup create + ``` ### Restoring GitLab from a backup @@ -626,16 +627,16 @@ released, you can update your GitLab instance: 1. SSH into your instance 1. Take a backup: - ```sh - sudo gitlab-rake gitlab:backup:create - ``` + ```sh + sudo gitlab-backup create + ``` 1. Update the repositories and install GitLab: - ```sh - sudo apt update - sudo apt install gitlab-ee - ``` + ```sh + sudo apt update + sudo apt install gitlab-ee + ``` After a few minutes, the new version should be up and running. diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index b1f79893baf..c5939dc6856 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -67,18 +67,19 @@ The first items we need to configure are the basic settings of the underlying vi 1. Enter a `User name` - e.g. **"gitlab-admin"** 1. Select an `Authentication type`, either **SSH public key** or **Password**: - > **Note:** if you're unsure which authentication type to use, select **Password** + > **Note:** if you're unsure which authentication type to use, select **Password** + + 1. If you chose **SSH public key** - enter your `SSH public key` into the field provided + _(read the [SSH documentation](../../ssh/README.md) to learn more about how to set up SSH + public keys)_ + 1. If you chose **Password** - enter the password you wish to use _(this is the password that you + will use later in this tutorial to [SSH] into the VM, so make sure it's a strong password/passphrase)_ - 1. If you chose **SSH public key** - enter your `SSH public key` into the field provided - _(read the [SSH documentation][GitLab-Docs-SSH] to learn more about how to set up SSH - public keys)_ - 1. If you chose **Password** - enter the password you wish to use _(this is the password that you - will use later in this tutorial to [SSH] into the VM, so make sure it's a strong password/passphrase)_ 1. Choose the appropriate `Subscription` tier for your Azure account 1. Choose an existing `Resource Group` or create a new one - e.g. **"GitLab-CE-Azure"** - > **Note:** a "Resource group" is a way to group related resources together for easier administration. - > We chose "GitLab-CE-Azure", but your resource group can have the same name as your VM. + > **Note:** a "Resource group" is a way to group related resources together for easier administration. + > We chose "GitLab-CE-Azure", but your resource group can have the same name as your VM. 1. Choose a `Location` - if you're unsure, select the default location @@ -248,6 +249,7 @@ rules in the list:  ## Connecting to GitLab + Use the domain name you set up earlier (or the public IP address) to visit your new GitLab instance in your browser. If everything has gone according to plan you should be presented with the following page, asking you to set a _new_ password for the administrator account automatically @@ -348,6 +350,7 @@ your VM, you can use the IP address in its place in the following command: ```bash ssh username@your-azure-domain-name.com ``` + Provide your password at the prompt to authenticate. #### SSH from Windows (PuTTY) @@ -404,25 +407,22 @@ on any cloud service you choose. ## Where to next? -Check out our other [Technical Articles][GitLab-Technical-Articles] or browse the [GitLab Documentation][GitLab-Docs] to learn more about GitLab. +Check out our other [Technical Articles](../../articles/index.md) or browse the [GitLab Documentation](../../README.md) to learn more about GitLab. ### Useful links - [GitLab Community Edition][CE] - [GitLab Enterprise Edition][EE] - [Microsoft Azure][Azure] - - [Azure - Free Account FAQ][Azure-Free-Account-FAQ] - - [Azure - Marketplace][Azure-Marketplace] - - [Azure Portal][Azure-Portal] - - [Azure - Pricing Calculator][Azure-Pricing-Calculator] - - [Azure - Troubleshoot SSH Connections to an Azure Linux VM][Azure-Troubleshoot-SSH-Connection] - - [Azure - Properly Shutdown an Azure VM][Azure-Properly-Shutdown-VM] + - [Azure - Free Account FAQ][Azure-Free-Account-FAQ] + - [Azure - Marketplace][Azure-Marketplace] + - [Azure Portal][Azure-Portal] + - [Azure - Pricing Calculator][Azure-Pricing-Calculator] + - [Azure - Troubleshoot SSH Connections to an Azure Linux VM][Azure-Troubleshoot-SSH-Connection] + - [Azure - Properly Shutdown an Azure VM][Azure-Properly-Shutdown-VM] - [SSH], [PuTTY] and [Using SSH in PuTTY][Using-SSH-In-Putty] [Original-Blog-Post]: https://about.gitlab.com/2016/07/13/how-to-setup-a-gitlab-instance-on-microsoft-azure/ "How to Set up a GitLab Instance on Microsoft Azure" -[GitLab-Docs]: https://docs.gitlab.com/ce/README.html "GitLab Documentation" -[GitLab-Technical-Articles]: https://docs.gitlab.com/ce/articles/index.html "GitLab Technical Articles" -[GitLab-Docs-SSH]: https://docs.gitlab.com/ce/ssh/README.html "GitLab Documentation: SSH" [CE]: https://about.gitlab.com/features/ [EE]: https://about.gitlab.com/features/#ee-starter diff --git a/doc/install/digitaloceandocker.md b/doc/install/digitaloceandocker.md index 63bb941ad47..b6bf7c95527 100644 --- a/doc/install/digitaloceandocker.md +++ b/doc/install/digitaloceandocker.md @@ -36,30 +36,30 @@ The rest of the steps are identical for macOS and Linux. 1. Login to Digital Ocean. 1. Generate a new API token at <https://cloud.digitalocean.com/settings/api/tokens>. - This command will create a new DO droplet called `gitlab-test-env-do` that will act as a docker host. + This command will create a new DO droplet called `gitlab-test-env-do` that will act as a docker host. - NOTE: **Note:** - 4GB is the minimum requirement for a Docker host that will run more than one GitLab instance. + NOTE: **Note:** + 4GB is the minimum requirement for a Docker host that will run more than one GitLab instance. - - RAM: 4GB - - Name: `gitlab-test-env-do` - - Driver: `digitalocean` + - RAM: 4GB + - Name: `gitlab-test-env-do` + - Driver: `digitalocean` 1. Set the DO token: - ```sh - export DOTOKEN=<your generated token> - ``` + ```sh + export DOTOKEN=<your generated token> + ``` 1. Create the machine: - ```sh - docker-machine create \ - --driver digitalocean \ - --digitalocean-access-token=$DOTOKEN \ - --digitalocean-size "4gb" \ - gitlab-test-env-do - ``` + ```sh + docker-machine create \ + --driver digitalocean \ + --digitalocean-access-token=$DOTOKEN \ + --digitalocean-size "4gb" \ + gitlab-test-env-do + ``` Resource: <https://docs.docker.com/machine/drivers/digital-ocean/>. diff --git a/doc/install/docker.md b/doc/install/docker.md index 06da65189ba..e0cef71a4d8 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -10,7 +10,7 @@ GitLab provides official Docker images allowing you to easily take advantage of ## Omnibus GitLab based images -GitLab maintains a set of [official Docker images](https://hub.docker.com/r/gitlab) based on our [Omnibus GitLab package](https://docs.gitlab.com/omnibus/README.html). These images include: +GitLab maintains a set of [official Docker images](https://hub.docker.com/u/gitlab) based on our [Omnibus GitLab package](https://docs.gitlab.com/omnibus/README.html). These images include: - [GitLab Community Edition](https://hub.docker.com/r/gitlab/gitlab-ce/) - [GitLab Enterprise Edition](https://hub.docker.com/r/gitlab/gitlab-ee/) diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index 14bf7012c01..be29bcc7cd7 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -30,16 +30,16 @@ To deploy GitLab on GCP you first need to create a virtual machine: 1. Go to <https://console.cloud.google.com/compute/instances> and log in with your Google credentials. 1. Click on **Create** -  +  -1. On the next page, you can select the type of VM as well as the +1. On the next page, you can select the type of VM as well as the estimated costs. Provide the name of the instance, desired datacenter, and machine type. Note that GitLab recommends at least 2 vCPU's and 4GB of RAM. -  +  1. Click **Change** under Boot disk to select the size, type, and desired operating system. GitLab supports a [variety of linux operating systems][req], including Ubuntu and Debian. Click **Select** when finished. -  +  1. As a last step allow HTTP and HTTPS traffic, then click **Create**. The process will finish in a few seconds. @@ -53,13 +53,13 @@ After a few seconds, the instance will be created and available to log in. The n 1. Click on the SSH button to connect to the instance. 1. A new window will appear, with you logged into the instance. -  +  1. Next, follow the instructions for installing GitLab for the operating system you choose, at <https://about.gitlab.com/install/>. You can use the IP address from the step above, as the hostname. 1. Congratulations! GitLab is now installed and you can access it via your browser. To finish installation, open the URL in your browser and provide the initial administrator password. The username for this account is `root`. -  +  ## Next steps @@ -83,31 +83,31 @@ here's how you configure GitLab to be aware of the change: 1. SSH into the VM. You can easily use the **SSH** button in the Google console and a new window will pop up. -  +  - In the future you might want to set up [connecting with an SSH key][ssh] - instead. + In the future you might want to set up [connecting with an SSH key][ssh] + instead. 1. Edit the config file of Omnibus GitLab using your favorite text editor: - ``` - sudo vim /etc/gitlab/gitlab.rb - ``` + ``` + sudo vim /etc/gitlab/gitlab.rb + ``` 1. Set the `external_url` value to the domain name you wish GitLab to have **without** `https`: - ``` - external_url 'http://gitlab.example.com' - ``` + ``` + external_url 'http://gitlab.example.com' + ``` - We will set up HTTPS in the next step, no need to do this now. + We will set up HTTPS in the next step, no need to do this now. 1. Reconfigure GitLab for the changes to take effect: - ``` - sudo gitlab-ctl reconfigure - ``` + ``` + sudo gitlab-ctl reconfigure + ``` 1. You can now visit GitLab using the domain name. diff --git a/doc/install/installation.md b/doc/install/installation.md index e9206469e5d..2989a4edaf7 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -6,7 +6,7 @@ type: howto This is the official installation guide to set up a production GitLab server using the source files. To set up a **development installation** or for many -other installation options, see the [main installation page](index.md). +other installation options, see the [main installation page](README.md). It was created for and tested on **Debian/Ubuntu** operating systems. Read [requirements.md](requirements.md) for hardware and operating system requirements. If you want to install on RHEL/CentOS, we recommend using the @@ -57,18 +57,18 @@ of this page: ``` - `/home/git/.ssh` - Contains OpenSSH settings. Specifically the `authorized_keys` - file managed by gitlab-shell. + file managed by GitLab Shell. - `/home/git/gitlab` - GitLab core software. - `/home/git/gitlab-shell` - Core add-on component of GitLab. Maintains SSH cloning and other functionality. - `/home/git/repositories` - Bare repositories for all projects organized by - namespace. This is where the git repositories which are pushed/pulled are + namespace. This is where the Git repositories which are pushed/pulled are maintained for all projects. **This area contains critical data for projects. [Keep a backup](../raketasks/backup_restore.md).** NOTE: **Note:** The default locations for repositories can be configured in `config/gitlab.yml` -of GitLab and `config.yml` of gitlab-shell. +of GitLab and `config.yml` of GitLab Shell. For a more in-depth overview, see the [GitLab architecture doc](../development/architecture.md). @@ -134,7 +134,7 @@ Make sure you have the right version of Git installed: # Install Git sudo apt-get install -y git-core -# Make sure Git is version 2.21.0 or higher +# Make sure Git is version 2.22.0 or higher git --version ``` @@ -171,9 +171,9 @@ sudo make install # Download and compile from source cd /tmp -curl --remote-name --location --progress https://www.kernel.org/pub/software/scm/git/git-2.21.0.tar.gz -echo '85eca51c7404da75e353eba587f87fea9481ba41e162206a6f70ad8118147bee git-2.21.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.21.0.tar.gz -cd git-2.21.0/ +curl --remote-name --location --progress https://www.kernel.org/pub/software/scm/git/git-2.22.0.tar.gz +echo 'a4b7e4365bee43caa12a38d646d2c93743d755d1cea5eab448ffb40906c9da0b git-2.22.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.22.0.tar.gz +cd git-2.22.0/ ./configure --with-libpcre make prefix=/usr/local all @@ -202,8 +202,8 @@ Then select 'Internet Site' and press enter to confirm the hostname. The Ruby interpreter is required to run GitLab. -**Note:** The current supported Ruby (MRI) version is 2.5.x. GitLab 11.6 - dropped support for Ruby 2.4.x. +**Note:** The current supported Ruby (MRI) version is 2.6.x. GitLab 12.2 + dropped support for Ruby 2.5.x. The use of Ruby version managers such as [RVM], [rbenv] or [chruby] with GitLab in production, frequently leads to hard to diagnose problems. For example, @@ -299,57 +299,57 @@ use of extensions and concurrent index removal, you need at least PostgreSQL 9.2 1. Install the database packages: - ```sh - sudo apt-get install -y postgresql postgresql-client libpq-dev postgresql-contrib - ``` + ```sh + sudo apt-get install -y postgresql postgresql-client libpq-dev postgresql-contrib + ``` 1. Create a database user for GitLab: - ```sh - sudo -u postgres psql -d template1 -c "CREATE USER git CREATEDB;" - ``` + ```sh + sudo -u postgres psql -d template1 -c "CREATE USER git CREATEDB;" + ``` 1. Create the `pg_trgm` extension (required for GitLab 8.6+): - ```sh - sudo -u postgres psql -d template1 -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;" - ``` + ```sh + sudo -u postgres psql -d template1 -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;" + ``` 1. Create the GitLab production database and grant all privileges on database: - ```sh - sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;" - ``` + ```sh + sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;" + ``` 1. Try connecting to the new database with the new user: - ```sh - sudo -u git -H psql -d gitlabhq_production - ``` + ```sh + sudo -u git -H psql -d gitlabhq_production + ``` 1. Check if the `pg_trgm` extension is enabled: - ```sh - SELECT true AS enabled - FROM pg_available_extensions - WHERE name = 'pg_trgm' - AND installed_version IS NOT NULL; - ``` + ```sh + SELECT true AS enabled + FROM pg_available_extensions + WHERE name = 'pg_trgm' + AND installed_version IS NOT NULL; + ``` - If the extension is enabled this will produce the following output: + If the extension is enabled this will produce the following output: - ``` - enabled - --------- - t - (1 row) - ``` + ``` + enabled + --------- + t + (1 row) + ``` 1. Quit the database session: - ```sh - gitlabhq_production> \q - ``` + ```sh + gitlabhq_production> \q + ``` ## 7. Redis @@ -569,7 +569,7 @@ GitLab Shell application startup time can be greatly reduced by disabling RubyGe - Compile Ruby with `configure --disable-rubygems` to disable RubyGems by default. Not recommended for system-wide Ruby. - Omnibus GitLab [replaces the *shebang* line of the `gitlab-shell/bin/*` scripts](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1707). -### Install gitlab-workhorse +### Install GitLab Workhorse GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/). The following command-line will install GitLab-Workhorse in `/home/git/gitlab-workhorse` @@ -789,7 +789,7 @@ nginx: configuration file /etc/nginx/nginx.conf test failed` sudo service nginx restart ``` -## Done! +## Post-install ### Double-check Application Status @@ -831,27 +831,28 @@ how to configure GitLab with a relative URL. To use GitLab with HTTPS: 1. In `gitlab.yml`: - 1. Set the `port` option in section 1 to `443`. - 1. Set the `https` option in section 1 to `true`. -1. In the `config.yml` of gitlab-shell: - 1. Set `gitlab_url` option to the HTTPS endpoint of GitLab (e.g. `https://git.example.com`). - 1. Set the certificates using either the `ca_file` or `ca_path` option. + 1. Set the `port` option in section 1 to `443`. + 1. Set the `https` option in section 1 to `true`. +1. In the `config.yml` of GitLab Shell: + 1. Set `gitlab_url` option to the HTTPS endpoint of GitLab (e.g. `https://git.example.com`). + 1. Set the certificates using either the `ca_file` or `ca_path` option. 1. Use the `gitlab-ssl` Nginx example config instead of the `gitlab` config. - 1. Update `YOUR_SERVER_FQDN`. - 1. Update `ssl_certificate` and `ssl_certificate_key`. - 1. Review the configuration file and consider applying other security and performance enhancing features. + 1. Update `YOUR_SERVER_FQDN`. + 1. Update `ssl_certificate` and `ssl_certificate_key`. + 1. Review the configuration file and consider applying other security and performance enhancing features. Using a self-signed certificate is discouraged but if you must use it, follow the normal directions. Then: 1. Generate a self-signed SSL certificate: - ```sh - mkdir -p /etc/nginx/ssl/ - cd /etc/nginx/ssl/ - sudo openssl req -newkey rsa:2048 -x509 -nodes -days 3560 -out gitlab.crt -keyout gitlab.key - sudo chmod o-r gitlab.key - ``` -1. In the `config.yml` of gitlab-shell set `self_signed_cert` to `true`. + ```sh + mkdir -p /etc/nginx/ssl/ + cd /etc/nginx/ssl/ + sudo openssl req -newkey rsa:2048 -x509 -nodes -days 3560 -out gitlab.crt -keyout gitlab.key + sudo chmod o-r gitlab.key + ``` + +1. In the `config.yml` of GitLab Shell set `self_signed_cert` to `true`. ### Enable Reply by email @@ -937,7 +938,7 @@ To use GitLab with Puma: cd /home/git/gitlab # Copy config file for the web server - sudo -u git -H config/puma.rb.example config/puma.rb + sudo -u git -H cp config/puma.rb.example config/puma.rb ``` 1. Edit the system `init.d` script to use `EXPERIMENTAL_PUMA=1` flag. If you have `/etc/default/gitlab`, then you should edit it instead. @@ -949,8 +950,8 @@ To use GitLab with Puma: If you see this message when attempting to clone a repository hosted by GitLab, this is likely due to an outdated Nginx or Apache configuration, or a missing or -misconfigured gitlab-workhorse instance. Double-check that you've -[installed Go](#3-go), [installed gitlab-workhorse](#install-gitlab-workhorse), +misconfigured GitLab Workhorse instance. Double-check that you've +[installed Go](#3-go), [installed GitLab Workhorse](#install-gitlab-workhorse), and correctly [configured Nginx](#site-configuration). ### google-protobuf "LoadError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found" diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index e4a2d9ecd68..fbbe2a34952 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -70,17 +70,17 @@ In short: 1. Open a terminal and in a new directory run: - ```sh - vagrant init openshift/origin-all-in-one - ``` + ```sh + vagrant init openshift/origin-all-in-one + ``` 1. This will generate a Vagrantfile based on the all-in-one VM image 1. In the same directory where you generated the Vagrantfile enter: - ```sh - vagrant up - ``` + ```sh + vagrant up + ``` This will download the VirtualBox image and fire up the VM with some preconfigured values as you can see in the Vagrantfile. As you may have noticed, you need @@ -195,22 +195,22 @@ In that case, the OpenShift service might not be running, so in order to fix it: 1. SSH into the VM by going to the directory where the Vagrantfile is and then run: - ```sh - vagrant ssh - ``` + ```sh + vagrant ssh + ``` 1. Run `systemctl` and verify by the output that the `openshift` service is not running (it will be in red color). If that's the case start the service with: - ```sh - sudo systemctl start openshift - ``` + ```sh + sudo systemctl start openshift + ``` 1. Verify the service is up with: - ```sh - systemctl status openshift -l - ``` + ```sh + systemctl status openshift -l + ``` Now you will be able to login using `oc` (like we did before) and visit the web console. @@ -393,55 +393,55 @@ Let's see how to do that using the following steps. 1. Make sure you are in the `gitlab` project: - ```sh - oc project gitlab - ``` + ```sh + oc project gitlab + ``` 1. See what services are used for this project: - ```sh - oc get svc - ``` + ```sh + oc get svc + ``` - The output will be similar to: + The output will be similar to: - ``` - NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE - gitlab-ce 172.30.243.177 <none> 22/TCP,80/TCP 5d - gitlab-ce-postgresql 172.30.116.75 <none> 5432/TCP 5d - gitlab-ce-redis 172.30.105.88 <none> 6379/TCP 5d - ``` + ``` + NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE + gitlab-ce 172.30.243.177 <none> 22/TCP,80/TCP 5d + gitlab-ce-postgresql 172.30.116.75 <none> 5432/TCP 5d + gitlab-ce-redis 172.30.105.88 <none> 6379/TCP 5d + ``` 1. We need to see the replication controllers of the `gitlab-ce` service. Get a detailed view of the current ones: - ```sh - oc describe rc gitlab-ce - ``` + ```sh + oc describe rc gitlab-ce + ``` - This will return a large detailed list of the current replication controllers. - Search for the name of the GitLab controller, usually `gitlab-ce-1` or if - that failed at some point and you spawned another one, it will be named - `gitlab-ce-2`. + This will return a large detailed list of the current replication controllers. + Search for the name of the GitLab controller, usually `gitlab-ce-1` or if + that failed at some point and you spawned another one, it will be named + `gitlab-ce-2`. 1. Scale GitLab using the previous information: - ```sh - oc scale --replicas=2 replicationcontrollers gitlab-ce-2 - ``` + ```sh + oc scale --replicas=2 replicationcontrollers gitlab-ce-2 + ``` 1. Get the new replicas number to make sure scaling worked: - ```sh - oc get rc gitlab-ce-2 - ``` + ```sh + oc get rc gitlab-ce-2 + ``` - which will return something like: + which will return something like: - ``` - NAME DESIRED CURRENT AGE - gitlab-ce-2 2 2 5d - ``` + ``` + NAME DESIRED CURRENT AGE + gitlab-ce-2 2 2 5d + ``` And that's it! We successfully scaled the replicas to 2 using the CLI. @@ -478,13 +478,13 @@ For OpenShift v3.0, you will need to do this manually: 1. Edit the Security Context: - ```sh - oc edit scc anyuid - ``` + ```sh + oc edit scc anyuid + ``` 1. Add `system:serviceaccount:<project>:gitlab-ce-user` to the `users` section. If you changed the Application Name from the default the user will - will be `<app-name>-user` instead of `gitlab-ce-user` + will be `<app-name>-user` instead of `gitlab-ce-user` 1. Save and exit the editor diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index b53624a33bf..595304e27e2 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -58,59 +58,59 @@ assumptions are made: Make sure to follow all steps below: -1. (Optional) If you run short on resources, you can temporarily free up some - memory by shutting down the GitLab service with the following command: +1. (Optional) If you run short on resources, you can temporarily free up some + memory by shutting down the GitLab service with the following command: - ```shell - sudo service gitlab stop - ``` + ```shell + sudo service gitlab stop + ``` -1. Create `/home/git/gitlab/config/initializers/relative_url.rb` +1. Create `/home/git/gitlab/config/initializers/relative_url.rb` - ```shell - cp /home/git/gitlab/config/initializers/relative_url.rb.sample \ - /home/git/gitlab/config/initializers/relative_url.rb - ``` + ```shell + cp /home/git/gitlab/config/initializers/relative_url.rb.sample \ + /home/git/gitlab/config/initializers/relative_url.rb + ``` - and change the following line: + and change the following line: - ```ruby - config.relative_url_root = "/gitlab" - ``` + ```ruby + config.relative_url_root = "/gitlab" + ``` -1. Edit `/home/git/gitlab/config/gitlab.yml` and uncomment/change the - following line: +1. Edit `/home/git/gitlab/config/gitlab.yml` and uncomment/change the + following line: - ```yaml - relative_url_root: /gitlab - ``` + ```yaml + relative_url_root: /gitlab + ``` -1. Edit `/home/git/gitlab/config/unicorn.rb` and uncomment/change the - following line: +1. Edit `/home/git/gitlab/config/unicorn.rb` and uncomment/change the + following line: - ```ruby - ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab" - ``` + ```ruby + ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab" + ``` -1. Edit `/home/git/gitlab-shell/config.yml` and append the relative path to - the following line: +1. Edit `/home/git/gitlab-shell/config.yml` and append the relative path to + the following line: - ```yaml - gitlab_url: http://127.0.0.1/gitlab - ``` + ```yaml + gitlab_url: http://127.0.0.1/gitlab + ``` -1. Make sure you have copied the supplied init script and the defaults file - as stated in the [installation guide](installation.md#install-init-script). - Then, edit `/etc/default/gitlab` and set in `gitlab_workhorse_options` the - `-authBackend` setting to read like: +1. Make sure you have copied the supplied init script and the defaults file + as stated in the [installation guide](installation.md#install-init-script). + Then, edit `/etc/default/gitlab` and set in `gitlab_workhorse_options` the + `-authBackend` setting to read like: - ```shell - -authBackend http://127.0.0.1:8080/gitlab - ``` + ```shell + -authBackend http://127.0.0.1:8080/gitlab + ``` - **Note:** - If you are using a custom init script, make sure to edit the above - gitlab-workhorse setting as needed. + **Note:** + If you are using a custom init script, make sure to edit the above + GitLab Workhorse setting as needed. 1. [Restart GitLab][] for the changes to take effect. @@ -118,9 +118,9 @@ Make sure to follow all steps below: To disable the relative URL: -1. Remove `/home/git/gitlab/config/initializers/relative_url.rb` +1. Remove `/home/git/gitlab/config/initializers/relative_url.rb` -1. Follow the same as above starting from 2. and set up the +1. Follow the same as above starting from 2. and set up the GitLab URL to one that doesn't contain a relative path. [omnibus-rel]: https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab "How to set up relative URL in Omnibus GitLab" diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 25ab608de3a..234e5acb394 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -40,7 +40,7 @@ Please consider using a virtual machine to run GitLab. ## Ruby versions -GitLab requires Ruby (MRI) 2.5. Support for Ruby versions below 2.5 (2.3, 2.4) will stop with GitLab 11.6. +GitLab requires Ruby (MRI) 2.6. Support for Ruby versions below 2.6 (2.4, 2.5) will stop with GitLab 12.2. You will have to use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://rubinius.com) but GitLab @@ -62,17 +62,19 @@ NOTE: **Note:** Since file system performance may affect GitLab's overall perfor ### CPU +This is the recommended minimum hardware for a handful of example GitLab user base sizes. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + - 1 core supports up to 100 users but the application can be a bit slower due to having all workers and background jobs running on the same core -- **2 cores** is the **recommended** number of cores and supports up to 500 users -- 4 cores supports up to 2,000 users -- 8 cores supports up to 5,000 users -- 16 cores supports up to 10,000 users -- 32 cores supports up to 20,000 users -- 64 cores supports up to 40,000 users -- More users? Run it on [multiple application servers](https://about.gitlab.com/high-availability/) +- **2 cores** is the **recommended** minimum number of cores and supports up to 100 users +- 4 cores supports up to 500 users +- 8 cores supports up to 1,000 users +- 32 cores supports up to 5,000 users +- More users? Run it high-availability on [multiple application servers](https://about.gitlab.com/high-availability/) ### Memory +This is the recommended minimum hardware for a handful of example GitLab user base sizes. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + You need at least 8GB of addressable memory (RAM + swap) to install and use GitLab! The operating system and any other running applications will also be using memory so keep in mind that you need at least 4GB available before running GitLab. With @@ -80,13 +82,11 @@ less memory GitLab will give strange errors during the reconfigure run and 500 errors during usage. - 4GB RAM + 4GB swap supports up to 100 users but it will be very slow -- **8GB RAM** is the **recommended** memory size for all installations and supports up to 100 users -- 16GB RAM supports up to 2,000 users -- 32GB RAM supports up to 4,000 users -- 64GB RAM supports up to 8,000 users -- 128GB RAM supports up to 16,000 users -- 256GB RAM supports up to 32,000 users -- More users? Run it on [multiple application servers](https://about.gitlab.com/high-availability/) +- **8GB RAM** is the **recommended** minimum memory size for all installations and supports up to 100 users +- 16GB RAM supports up to 500 users +- 32GB RAM supports up to 1,000 users +- 128GB RAM supports up to 5,000 users +- More users? Run it high-availability on [multiple application servers](https://about.gitlab.com/high-availability/) We recommend having at least [2GB of swap on your server](https://askubuntu.com/a/505344/310789), even if you currently have enough available RAM. Having swap will help reduce the chance of errors occurring @@ -94,6 +94,8 @@ if your available memory changes. We also recommend [configuring the kernel's sw to a low value like `10` to make the most of your RAM while still having the swap available when needed. +Our [Memory Team](https://about.gitlab.com/handbook/engineering/development/enablement/memory/) is actively working to reduce the memory requirement. + NOTE: **Note:** The 25 workers of Sidekiq will show up as separate processes in your process overview (such as `top` or `htop`) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. ## Database @@ -146,8 +148,8 @@ CREATE EXTENSION postgres_fdw; ## Unicorn Workers -For most instances we recommend using: CPU cores + 1 = unicorn workers. -So for a machine with 2 cores, 3 unicorn workers is ideal. +For most instances we recommend using: (CPU cores * 1.5) + 1 = unicorn workers. +For example a node with 4 cores would have 7 unicorn workers. For all machines that have 2GB and up we recommend a minimum of three unicorn workers. If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive swapping. diff --git a/doc/integration/README.md b/doc/integration/README.md index 135952a1b08..55f9666e3a3 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -27,7 +27,7 @@ See the documentation below for details on how to configure these services. - [SAML](saml.md) Configure GitLab as a SAML 2.0 Service Provider - [Trello](trello_power_up.md) Integrate Trello with GitLab -> GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. +> GitLab Enterprise Edition contains [advanced Jenkins support](jenkins.md). ## Project services @@ -70,5 +70,3 @@ After that restart GitLab with: ```bash sudo gitlab-ctl restart ``` - -[jenkins]: https://docs.gitlab.com/ee/integration/jenkins.html diff --git a/doc/integration/auth0.md b/doc/integration/auth0.md index c67375ede50..2dd6fa3d5a2 100644 --- a/doc/integration/auth0.md +++ b/doc/integration/auth0.md @@ -16,64 +16,64 @@ application. 1. At the top of the Settings screen, you should see your Domain, Client ID and Client Secret. Take note of these as you'll need to put them in the configuration file. For example: - - Domain: `test1234.auth0.com` - - Client ID: `t6X8L2465bNePWLOvt9yi41i` - - Client Secret: `KbveM3nqfjwCbrhaUy_gDu2dss8TIlHIdzlyf33pB7dEK5u_NyQdp65O_o02hXs2` + - Domain: `test1234.auth0.com` + - Client ID: `t6X8L2465bNePWLOvt9yi41i` + - Client Secret: `KbveM3nqfjwCbrhaUy_gDu2dss8TIlHIdzlyf33pB7dEK5u_NyQdp65O_o02hXs2` 1. Fill in the Allowed Callback URLs: - - `http://YOUR_GITLAB_URL/users/auth/auth0/callback` (or) - - `https://YOUR_GITLAB_URL/users/auth/auth0/callback` + - `http://YOUR_GITLAB_URL/users/auth/auth0/callback` (or) + - `https://YOUR_GITLAB_URL/users/auth/auth0/callback` 1. Fill in the Allowed Origins (CORS): - - `http://YOUR_GITLAB_URL` (or) - - `https://YOUR_GITLAB_URL` + - `http://YOUR_GITLAB_URL` (or) + - `https://YOUR_GITLAB_URL` 1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. 1. Add the provider configuration: - For omnibus package: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "auth0", - "args" => { client_id: 'YOUR_AUTH0_CLIENT_ID', - client_secret: 'YOUR_AUTH0_CLIENT_SECRET', - domain: 'YOUR_AUTH0_DOMAIN', - scope: 'openid profile email' - } - } - ] - ``` - - For installations from source: - - ```yaml - - { name: 'auth0', - args: { - client_id: 'YOUR_AUTH0_CLIENT_ID', - client_secret: 'YOUR_AUTH0_CLIENT_SECRET', - domain: 'YOUR_AUTH0_DOMAIN', - scope: 'openid profile email' } - } - ``` + For Omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "auth0", + "args" => { client_id: 'YOUR_AUTH0_CLIENT_ID', + client_secret: 'YOUR_AUTH0_CLIENT_SECRET', + domain: 'YOUR_AUTH0_DOMAIN', + scope: 'openid profile email' + } + } + ] + ``` + + For installations from source: + + ```yaml + - { name: 'auth0', + args: { + client_id: 'YOUR_AUTH0_CLIENT_ID', + client_secret: 'YOUR_AUTH0_CLIENT_SECRET', + domain: 'YOUR_AUTH0_DOMAIN', + scope: 'openid profile email' } + } + ``` 1. Change `YOUR_AUTH0_CLIENT_ID` to the client ID from the Auth0 Console page from step 5. @@ -81,8 +81,8 @@ application. 1. Change `YOUR_AUTH0_CLIENT_SECRET` to the client secret from the Auth0 Console page from step 5. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be an Auth0 icon below the regular sign in form. Click the icon to begin the authentication process. Auth0 will ask the diff --git a/doc/integration/azure.md b/doc/integration/azure.md index a9468f201ef..c30d79e3dab 100644 --- a/doc/integration/azure.md +++ b/doc/integration/azure.md @@ -32,7 +32,7 @@ To enable the Microsoft Azure OAuth2 OmniAuth provider you must register your ap 1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: ```sh sudo editor /etc/gitlab/gitlab.rb @@ -50,7 +50,7 @@ To enable the Microsoft Azure OAuth2 OmniAuth provider you must register your ap 1. Add the provider configuration: - For omnibus package: + For Omnibus package: ```ruby gitlab_rails['omniauth_providers'] = [ diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md index 68ec8c4b5c2..5d8f2ebcb8b 100644 --- a/doc/integration/bitbucket.md +++ b/doc/integration/bitbucket.md @@ -30,97 +30,97 @@ To enable the Bitbucket OmniAuth provider you must register your application with Bitbucket.org. Bitbucket will generate an application ID and secret key for you to use. -1. Sign in to [Bitbucket.org](https://bitbucket.org). -1. Navigate to your individual user settings (**Bitbucket settings**) or a team's - settings (**Manage team**), depending on how you want the application registered. - It does not matter if the application is registered as an individual or a - team, that is entirely up to you. -1. Select **OAuth** in the left menu under "Access Management". -1. Select **Add consumer**. -1. Provide the required details: - - | Item | Description | - | :--- | :---------- | - | **Name** | This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. | - | **Application description** | Fill this in if you wish. | - | **Callback URL** | The URL to your GitLab installation, e.g., `https://gitlab.example.com/users/auth`. | - | **URL** | The URL to your GitLab installation, e.g., `https://gitlab.example.com`. | - - NOTE: Be sure to append `/users/auth` to the end of the callback URL - to prevent a [OAuth2 convert - redirect](http://tetraph.com/covert_redirect/) vulnerability. - - NOTE: Starting in GitLab 8.15, you MUST specify a callback URL, or you will - see an "Invalid redirect_uri" message. For more details, see [the - Bitbucket documentation](https://confluence.atlassian.com/bitbucket/oauth-faq-338365710.html). - - And grant at least the following permissions: - - ``` - Account: Email, Read - Projects: Read - Repositories: Read - Pull Requests: Read - Issues: Read - Wiki: Read and Write - ``` - -  - -1. Select **Save**. -1. Select your newly created OAuth consumer and you should now see a Key and - Secret in the list of OAuth consumers. Keep this page open as you continue - the configuration. - -  - -1. On your GitLab server, open the configuration file: - - ``` - # For Omnibus packages - sudo editor /etc/gitlab/gitlab.rb - - # For installations from source - sudo -u git -H editor /home/git/gitlab/config/gitlab.yml - ``` - -1. Add the Bitbucket provider configuration: - - For Omnibus packages: - - ```ruby - gitlab_rails['omniauth_enabled'] = true - - gitlab_rails['omniauth_providers'] = [ - { - "name" => "bitbucket", - "app_id" => "BITBUCKET_APP_KEY", - "app_secret" => "BITBUCKET_APP_SECRET", - "url" => "https://bitbucket.org/" - } - ] - ``` - - For installations from source: - - ```yaml - omniauth: - enabled: true - providers: - - { name: 'bitbucket', - app_id: 'BITBUCKET_APP_KEY', - app_secret: 'BITBUCKET_APP_SECRET', - url: 'https://bitbucket.org/' } - ``` - - --- - - Where `BITBUCKET_APP_KEY` is the Key and `BITBUCKET_APP_SECRET` the Secret - from the Bitbucket application page. - -1. Save the configuration file. -1. For the changes to take effect, [reconfigure GitLab][] if you installed via - Omnibus, or [restart][] if installed from source. +1. Sign in to [Bitbucket.org](https://bitbucket.org). +1. Navigate to your individual user settings (**Bitbucket settings**) or a team's + settings (**Manage team**), depending on how you want the application registered. + It does not matter if the application is registered as an individual or a + team, that is entirely up to you. +1. Select **OAuth** in the left menu under "Access Management". +1. Select **Add consumer**. +1. Provide the required details: + + | Item | Description | + | :--- | :---------- | + | **Name** | This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. | + | **Application description** | Fill this in if you wish. | + | **Callback URL** | The URL to your GitLab installation, e.g., `https://gitlab.example.com/users/auth`. | + | **URL** | The URL to your GitLab installation, e.g., `https://gitlab.example.com`. | + + NOTE: Be sure to append `/users/auth` to the end of the callback URL + to prevent a [OAuth2 convert + redirect](http://tetraph.com/covert_redirect/) vulnerability. + + NOTE: Starting in GitLab 8.15, you MUST specify a callback URL, or you will + see an "Invalid redirect_uri" message. For more details, see [the + Bitbucket documentation](https://confluence.atlassian.com/bitbucket/oauth-faq-338365710.html). + + And grant at least the following permissions: + + ``` + Account: Email, Read + Projects: Read + Repositories: Read + Pull Requests: Read + Issues: Read + Wiki: Read and Write + ``` + +  + +1. Select **Save**. +1. Select your newly created OAuth consumer and you should now see a Key and + Secret in the list of OAuth consumers. Keep this page open as you continue + the configuration. + +  + +1. On your GitLab server, open the configuration file: + + ``` + # For Omnibus packages + sudo editor /etc/gitlab/gitlab.rb + + # For installations from source + sudo -u git -H editor /home/git/gitlab/config/gitlab.yml + ``` + +1. Add the Bitbucket provider configuration: + + For Omnibus packages: + + ```ruby + gitlab_rails['omniauth_enabled'] = true + + gitlab_rails['omniauth_providers'] = [ + { + "name" => "bitbucket", + "app_id" => "BITBUCKET_APP_KEY", + "app_secret" => "BITBUCKET_APP_SECRET", + "url" => "https://bitbucket.org/" + } + ] + ``` + + For installations from source: + + ```yaml + omniauth: + enabled: true + providers: + - { name: 'bitbucket', + app_id: 'BITBUCKET_APP_KEY', + app_secret: 'BITBUCKET_APP_SECRET', + url: 'https://bitbucket.org/' } + ``` + + --- + + Where `BITBUCKET_APP_KEY` is the Key and `BITBUCKET_APP_SECRET` the Secret + from the Bitbucket application page. + +1. Save the configuration file. +1. For the changes to take effect, [reconfigure GitLab][] if you installed via + Omnibus, or [restart][] if installed from source. On the sign in page there should now be a Bitbucket icon below the regular sign in form. Click the icon to begin the authentication process. Bitbucket will ask diff --git a/doc/integration/cas.md b/doc/integration/cas.md index c6178fa44f0..83b64773c9f 100644 --- a/doc/integration/cas.md +++ b/doc/integration/cas.md @@ -2,63 +2,63 @@ To enable the CAS OmniAuth provider you must register your application with your CAS instance. This requires the service URL GitLab will supply to CAS. It should be something like: `https://gitlab.example.com:443/users/auth/cas3/callback?url`. By default handling for SLO is enabled, you only need to configure CAS for backchannel logout. -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration: +1. Add the provider configuration: - For omnibus package: + For Omnibus package: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name"=> "cas3", - "label"=> "cas", - "args"=> { - "url"=> 'CAS_SERVER', - "login_url"=> '/CAS_PATH/login', - "service_validate_url"=> '/CAS_PATH/p3/serviceValidate', - "logout_url"=> '/CAS_PATH/logout' - } - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name"=> "cas3", + "label"=> "cas", + "args"=> { + "url"=> 'CAS_SERVER', + "login_url"=> '/CAS_PATH/login', + "service_validate_url"=> '/CAS_PATH/p3/serviceValidate', + "logout_url"=> '/CAS_PATH/logout' + } + } + ] + ``` - For installations from source: + For installations from source: - ``` - - { name: 'cas3', - label: 'cas', - args: { - url: 'CAS_SERVER', - login_url: '/CAS_PATH/login', - service_validate_url: '/CAS_PATH/p3/serviceValidate', - logout_url: '/CAS_PATH/logout'} } - ``` + ``` + - { name: 'cas3', + label: 'cas', + args: { + url: 'CAS_SERVER', + login_url: '/CAS_PATH/login', + service_validate_url: '/CAS_PATH/p3/serviceValidate', + logout_url: '/CAS_PATH/logout'} } + ``` -1. Change 'CAS_PATH' to the root of your CAS instance (ie. `cas`). +1. Change 'CAS_PATH' to the root of your CAS instance (ie. `cas`). -1. If your CAS instance does not use default TGC lifetimes, update the `cas3.session_duration` to at least the current TGC maximum lifetime. To explicitly disable SLO, regardless of CAS settings, set this to 0. +1. If your CAS instance does not use default TGC lifetimes, update the `cas3.session_duration` to at least the current TGC maximum lifetime. To explicitly disable SLO, regardless of CAS settings, set this to 0. -1. Save the configuration file. +1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a CAS tab in the sign in form. diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index da1df07a75d..de49508b47a 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -12,6 +12,7 @@ special searches: - [Advanced Syntax Search](../user/search/advanced_search_syntax.md) ## Version Requirements + <!-- Please remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes --> | GitLab version | Elasticsearch version | @@ -23,25 +24,32 @@ special searches: ## Installing Elasticsearch Elasticsearch is _not_ included in the Omnibus packages. You will have to -install it yourself whether you are using the Omnibus package or installed -GitLab from source. Providing detailed information on installing Elasticsearch -is out of the scope of this document. +[install it yourself](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html "Elasticsearch installation documentation") +whether you are using the Omnibus package or installed GitLab from source. +Providing detailed information on installing Elasticsearch is out of the scope +of this document. + +NOTE: **Note:** +Elasticsearch should be installed on a separate server, whether you install +it yourself or by using the +[Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) +service. Running Elasticsearch on the same server as GitLab is not recommended +and it will likely cause performance degradation on the GitLab installation. Once the data is added to the database or repository and [Elasticsearch is enabled in the admin area](#enabling-elasticsearch) the search index will be -updated automatically. Elasticsearch can be installed on the same machine as -GitLab or on a separate server, or you can use the [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) -service. - -You can follow the steps as described in the [official web site](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html "Elasticsearch installation documentation") or -use the packages that are available for your OS. +updated automatically. ## Elasticsearch repository indexer (beta) In order to improve elasticsearch indexing performance, GitLab has made available a [new indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). This will replace the included Ruby indexer in the future but should be considered beta software for now, so there may be some bugs. -If you would like to use it, please follow the instructions below. +The Elasticsearch Go indexer is included in Omnibus for GitLab 11.8 and newer. + +To use the new Elasticsearch indexer included in Omnibus, check the box "Use the new repository indexer (beta)" when [enabling the Elasticsearch integration](#enabling-elasticsearch). + +If you would like to use the Elasticsearch Go indexer with a source installation or an older version of GitLab, please follow the instructions below. ### Installation @@ -115,8 +123,8 @@ production instances, they recommend considerably more resources. Storage requirements also vary based on the installation side, but as a rule of thumb, you should allocate the total size of your production database, **plus** -two-thirds of the total size of your git repositories. Efforts to reduce this -total are being tracked in this epic: [gitlab-org&153](https://gitlab.com/groups/gitlab-org/-/epics/153). +two-thirds of the total size of your Git repositories. Efforts to reduce this +total are being tracked in [epic &153](https://gitlab.com/groups/gitlab-org/-/epics/153). ## Enabling Elasticsearch @@ -325,30 +333,36 @@ curl --request PUT localhost:9200/gitlab-production/_settings --data '{ Enable Elasticsearch search in **Admin > Settings > Integrations**. That's it. Enjoy it! +### Index limit + +Currently for repository and snippet files, GitLab would only index up to 1 MB of content, in order to avoid indexing timeout. + ## GitLab Elasticsearch Rake Tasks There are several rake tasks available to you via the command line: -- [sudo gitlab-rake gitlab:elastic:index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This is a wrapper task. It does the following: - `sudo gitlab-rake gitlab:elastic:create_empty_index` - `sudo gitlab-rake gitlab:elastic:clear_index_status` - `sudo gitlab-rake gitlab:elastic:index_projects` - `sudo gitlab-rake gitlab:elastic:index_snippets` -- [sudo gitlab-rake gitlab:elastic:index_projects](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This iterates over all projects and queues sidekiq jobs to index them in the background. -- [sudo gitlab-rake gitlab:elastic:index_projects_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. -- [sudo gitlab-rake gitlab:elastic:create_empty_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:create_empty_index`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This generates an empty index on the Elasticsearch side. -- [sudo gitlab-rake gitlab:elastic:clear_index_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This deletes all instances of IndexStatus for all projects. -- [sudo gitlab-rake gitlab:elastic:delete_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:delete_index`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - This removes the GitLab index on the Elasticsearch instance. -- [sudo gitlab-rake gitlab:elastic:recreate_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:recreate_index`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - Does the same thing as `sudo gitlab-rake gitlab:elastic:create_empty_index` -- [sudo gitlab-rake gitlab:elastic:index_snippets](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) +- [`sudo gitlab-rake gitlab:elastic:index_snippets`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) - Performs an Elasticsearch import that indexes the snippets data. +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Displays which projects are not indexed. ### Environment Variables @@ -424,91 +438,94 @@ Here are some common pitfalls and how to overcome them: - **How can I verify my GitLab instance is using Elasticsearch?** - The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: + The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: - ```ruby - u = User.find_by_username('your-username') - s = SearchService.new(u, {:search => 'search_term'}) - pp s.search_objects.class.name - ``` + ```ruby + u = User.find_by_username('your-username') + s = SearchService.new(u, {:search => 'search_term'}) + pp s.search_objects.class.name + ``` - If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. + If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. - **I updated GitLab and now I can't find anything** - We continuously make updates to our indexing strategies and aim to support - newer versions of Elasticsearch. When indexing changes are made, it may - be necessary for you to [reindex](#adding-gitlabs-data-to-the-elasticsearch-index) after updating GitLab. + We continuously make updates to our indexing strategies and aim to support + newer versions of Elasticsearch. When indexing changes are made, it may + be necessary for you to [reindex](#adding-gitlabs-data-to-the-elasticsearch-index) after updating GitLab. - **I indexed all the repositories but I can't find anything** - Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). + Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). - Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. + Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. - If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): + If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): - ```ruby - u = User.find_by_username('your-username') - s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’}) - pp s.search_objects.to_a - ``` + ```ruby + u = User.find_by_username('your-username') + s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’}) + pp s.search_objects.to_a + ``` - See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data. + See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data. - **I indexed all the repositories but then switched Elasticsearch servers and now I can't find anything** - You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. + You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. - **The indexing process is taking a very long time** - The more data present in your GitLab instance, the longer the indexing process takes. + The more data present in your GitLab instance, the longer the indexing process takes. + +- **There are some projects that weren't indexed, but we don't know which ones** + + You can run `sudo gitlab-rake gitlab:elastic:projects_not_indexed` to display projects that aren't indexed. - **No new data is added to the Elasticsearch index when I push code** - When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could - happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, - run the `gitlab:elastic:clear_locked_projects` rake task. + When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could + happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, + run the `gitlab:elastic:clear_locked_projects` rake task. - **"Can't specify parent if no parent field has been configured"** - If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get - exception in lots of different cases: - - ```text - Elasticsearch::Transport::Transport::Errors::BadRequest([400] { - "error": { - "root_cause": [{ - "type": "illegal_argument_exception", - "reason": "Can't specify parent if no parent field has been configured" - }], - "type": "illegal_argument_exception", - "reason": "Can't specify parent if no parent field has been configured" - }, - "status": 400 - }): - ``` - - This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, - see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch). + If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get + exception in lots of different cases: + + ```text + Elasticsearch::Transport::Transport::Errors::BadRequest([400] { + "error": { + "root_cause": [{ + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }], + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }, + "status": 400 + }): + ``` + + This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, + see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch). - Exception `Elasticsearch::Transport::Transport::Errors::BadRequest` - If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#system-requirements). - There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command. + If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#system-requirements). + There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command. - Exception `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge` - ```text - [413] {"Message":"Request size exceeded 10485760 bytes"} - ``` - - This exception is seen when your Elasticsearch cluster is configured to reject - requests above a certain size (10MiB in this case). This corresponds to the - `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a - larger size and restart your Elasticsearch cluster. + ```text + [413] {"Message":"Request size exceeded 10485760 bytes"} + ``` - AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html) - for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of - the underlying instance. + This exception is seen when your Elasticsearch cluster is configured to reject + requests above a certain size (10MiB in this case). This corresponds to the + `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a + larger size and restart your Elasticsearch cluster. + AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html) + for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of + the underlying instance. diff --git a/doc/integration/facebook.md b/doc/integration/facebook.md index fe789a80eed..d46486ad888 100644 --- a/doc/integration/facebook.md +++ b/doc/integration/facebook.md @@ -2,7 +2,7 @@ To enable the Facebook OmniAuth provider you must register your application with Facebook. Facebook will generate an app ID and secret key for you to use. -1. Sign in to the [Facebook Developer Platform](https://developers.facebook.com/). +1. Sign in to the [Facebook Developer Platform](https://developers.facebook.com/). 1. Choose "My Apps" > "Add a New App" @@ -19,7 +19,7 @@ To enable the Facebook OmniAuth provider you must register your application with 1. Enter the address of your GitLab installation at the bottom of the package -  +  1. Choose "Next" @@ -29,7 +29,7 @@ To enable the Facebook OmniAuth provider you must register your application with 1. Fill in a contact email for your app -  +  1. Choose "Save Changes" @@ -45,55 +45,55 @@ To enable the Facebook OmniAuth provider you must register your application with 1. You should now see an app key and app secret (see screenshot). Keep this page open as you continue configuration. -  +  -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration: +1. Add the provider configuration: - For omnibus package: + For Omnibus package: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "facebook", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET" - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "facebook", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET" + } + ] + ``` - For installations from source: + For installations from source: - ``` - - { name: 'facebook', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET' } - ``` + ``` + - { name: 'facebook', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } + ``` -1. Change 'YOUR_APP_ID' to the API key from Facebook page in step 10. +1. Change 'YOUR_APP_ID' to the API key from Facebook page in step 10. -1. Change 'YOUR_APP_SECRET' to the API secret from the Facebook page in step 10. +1. Change 'YOUR_APP_SECRET' to the API secret from the Facebook page in step 10. -1. Save the configuration file. +1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a Facebook icon below the regular sign in form. Click the icon to begin the authentication process. Facebook will ask the user to sign in and authorize the GitLab application. If everything goes well the user will be returned to GitLab and will be signed in. diff --git a/doc/integration/github.md b/doc/integration/github.md index 5b01dd9feb7..f19b3109d15 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -7,111 +7,111 @@ You can integrate your GitLab instance with GitHub.com as well as GitHub Enterpr To enable GitHub OmniAuth provider, you must use GitHub's credentials for your GitLab instance. To get the credentials (a pair of Client ID and Client Secret), you must register an application as an OAuth App on GitHub. -1. Sign in to GitHub. +1. Sign in to GitHub. -1. Navigate to your individual user or organization settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or an organization - that is entirely up to you. +1. Navigate to your individual user or organization settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or an organization - that is entirely up to you. - - For individual accounts, select **Developer settings** from the left menu, then select **OAuth Apps**. - - For organization accounts, directly select **OAuth Apps** from the left menu. + - For individual accounts, select **Developer settings** from the left menu, then select **OAuth Apps**. + - For organization accounts, directly select **OAuth Apps** from the left menu. -1. Select **Register an application** (if you don't have any OAuth App) or **New OAuth App** (if you already have OAuth Apps). -  +1. Select **Register an application** (if you don't have any OAuth App) or **New OAuth App** (if you already have OAuth Apps). +  -1. Provide the required details. - - Application name: This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. - - Homepage URL: The URL of your GitLab installation. For example, `https://gitlab.example.com`. - - Application description: Fill this in if you wish. - - Authorization callback URL: `http(s)://${YOUR_DOMAIN}/users/auth`. Please make sure the port is included if your GitLab instance is not configured on default port. -  +1. Provide the required details. + - Application name: This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. + - Homepage URL: The URL of your GitLab installation. For example, `https://gitlab.example.com`. + - Application description: Fill this in if you wish. + - Authorization callback URL: `http(s)://${YOUR_DOMAIN}/users/auth`. Please make sure the port is included if your GitLab instance is not configured on default port. +  - NOTE: Be sure to append `/users/auth` to the end of the callback URL - to prevent a [OAuth2 convert - redirect](http://tetraph.com/covert_redirect/) vulnerability. + NOTE: Be sure to append `/users/auth` to the end of the callback URL + to prevent a [OAuth2 convert + redirect](http://tetraph.com/covert_redirect/) vulnerability. -1. Select **Register application**. +1. Select **Register application**. -1. You should now see a pair of **Client ID** and **Client Secret** near the top right of the page (see screenshot). - Keep this page open as you continue configuration. -  +1. You should now see a pair of **Client ID** and **Client Secret** near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration: +1. Add the provider configuration: - For omnibus package: + For Omnibus package: - For GitHub.com: + For GitHub.com: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "github", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "scope" => "user:email" } - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "github", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "scope" => "user:email" } + } + ] + ``` - For GitHub Enterprise: + For GitHub Enterprise: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "github", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "url" => "https://github.example.com/", - "args" => { "scope" => "user:email" } - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "github", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://github.example.com/", + "args" => { "scope" => "user:email" } + } + ] + ``` - For installation from source: + For installation from source: - For GitHub.com: + For GitHub.com: - ``` - - { name: 'github', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { scope: 'user:email' } } - ``` + ``` + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'user:email' } } + ``` - For GitHub Enterprise: + For GitHub Enterprise: - ``` - - { name: 'github', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - url: "https://github.example.com/", - args: { scope: 'user:email' } } - ``` + ``` + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + args: { scope: 'user:email' } } + ``` - __Replace `https://github.example.com/` with your GitHub URL.__ + __Replace `https://github.example.com/` with your GitHub URL.__ -1. Change `YOUR_APP_ID` to the Client ID from the GitHub application page from step 6. +1. Change `YOUR_APP_ID` to the Client ID from the GitHub application page from step 6. -1. Change `YOUR_APP_SECRET` to the Client Secret from the GitHub application page from step 6. +1. Change `YOUR_APP_SECRET` to the Client Secret from the GitHub application page from step 6. -1. Save the configuration file. +1. Save the configuration file. -1. [Reconfigure GitLab][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. [Reconfigure GitLab][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a GitHub icon below the regular sign in form. Click the icon to begin the authentication process. GitHub will ask the user to sign in and authorize the GitLab application. @@ -124,19 +124,19 @@ certificate and the imports are failing, you will need to disable SSL verificati It should be disabled by adding `verify_ssl` to `false` in the provider configuration and changing the global Git `sslVerify` option to `false` in the GitLab server. -For omnibus package: +For Omnibus package: ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "github", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "url" => "https://github.example.com/", - "verify_ssl" => false, - "args" => { "scope" => "user:email" } - } - ] +gitlab_rails['omniauth_providers'] = [ + { + "name" => "github", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "url" => "https://github.example.com/", + "verify_ssl" => false, + "args" => { "scope" => "user:email" } + } +] ``` You will also need to disable Git SSL verification on the server hosting GitLab. @@ -148,11 +148,11 @@ omnibus_gitconfig['system'] = { "http" => ["sslVerify = false"] } For installation from source: ``` - - { name: 'github', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - url: "https://github.example.com/", - verify_ssl: false, - args: { scope: 'user:email' } } +- { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + verify_ssl: false, + args: { scope: 'user:email' } } ``` You will also need to disable Git SSL verification on the server hosting GitLab. diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 70087576678..74f2d5cb403 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -5,78 +5,78 @@ Import projects from GitLab.com and login to your GitLab instance with your GitL To enable the GitLab.com OmniAuth provider you must register your application with GitLab.com. GitLab.com will generate an application ID and secret key for you to use. -1. Sign in to GitLab.com +1. Sign in to GitLab.com 1. On the upper right corner, click on your avatar and go to your **Settings**. -1. Select **Applications** in the left menu. +1. Select **Applications** in the left menu. -1. Provide the required details for **Add new application**. - - Name: This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. - - Redirect URI: +1. Provide the required details for **Add new application**. + - Name: This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. + - Redirect URI: - ``` - http://your-gitlab.example.com/import/gitlab/callback - http://your-gitlab.example.com/users/auth/gitlab/callback - ``` + ``` + http://your-gitlab.example.com/import/gitlab/callback + http://your-gitlab.example.com/users/auth/gitlab/callback + ``` - The first link is required for the importer and second for the authorization. + The first link is required for the importer and second for the authorization. -1. Select **Save application**. +1. Select **Save application**. -1. You should now see a **Application Id** and **Secret** near the top right of the page (see screenshot). - Keep this page open as you continue configuration. -  +1. You should now see a **Application Id** and **Secret** near the top right of the page (see screenshot). + Keep this page open as you continue configuration. +  -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration: +1. Add the provider configuration: - For omnibus package: + For Omnibus package: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "gitlab", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "scope" => "api" } - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "gitlab", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "scope" => "api" } + } + ] + ``` - For installations from source: + For installations from source: - ``` - - { name: 'gitlab', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { scope: 'api' } } - ``` + ``` + - { name: 'gitlab', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'api' } } + ``` -1. Change 'YOUR_APP_ID' to the Application ID from the GitLab.com application page. +1. Change 'YOUR_APP_ID' to the Application ID from the GitLab.com application page. -1. Change 'YOUR_APP_SECRET' to the secret from the GitLab.com application page. +1. Change 'YOUR_APP_SECRET' to the secret from the GitLab.com application page. -1. Save the configuration file. +1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a GitLab.com icon below the regular sign in form. Click the icon to begin the authentication process. GitLab.com will ask the user to sign in and authorize the GitLab application. diff --git a/doc/integration/google.md b/doc/integration/google.md index d2b4e119978..4f6999571b6 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -10,10 +10,10 @@ In Google's side: 1. Navigate to the [cloud resource manager](https://console.cloud.google.com/cloud-resource-manager) page 1. Select **Create Project** 1. Provide the project information: - - **Project name** - "GitLab" works just fine here. - - **Project ID** - Must be unique to all Google Developer registered applications. - Google provides a randomly generated Project ID by default. You can use - the randomly generated ID or choose a new one. + - **Project name** - "GitLab" works just fine here. + - **Project ID** - Must be unique to all Google Developer registered applications. + Google provides a randomly generated Project ID by default. You can use + the randomly generated ID or choose a new one. 1. Refresh the page and you should see your new project in the list 1. Go to the [Google API Console](https://console.developers.google.com/apis/dashboard) 1. Select the previously created project form the upper left corner @@ -21,17 +21,17 @@ In Google's side: 1. Select **OAuth consent screen** and fill the form with the required information 1. In the **Credentials** tab, select **Create credentials > OAuth client ID** 1. Fill in the required information - - **Application type** - Choose "Web Application" - - **Name** - Use the default one or provide your own - - **Authorized JavaScript origins** -This isn't really used by GitLab but go - ahead and put `https://gitlab.example.com` - - **Authorized redirect URIs** - Enter your domain name followed by the - callback URIs one at a time: - - ``` - https://gitlab.example.com/users/auth/google_oauth2/callback - https://gitlab.example.com/-/google_api/auth/callback - ``` + - **Application type** - Choose "Web Application" + - **Name** - Use the default one or provide your own + - **Authorized JavaScript origins** -This isn't really used by GitLab but go + ahead and put `https://gitlab.example.com` + - **Authorized redirect URIs** - Enter your domain name followed by the + callback URIs one at a time: + + ``` + https://gitlab.example.com/users/auth/google_oauth2/callback + https://gitlab.example.com/-/google_api/auth/callback + ``` 1. You should now be able to see a Client ID and Client secret. Note them down or keep this page open as you will need them later. @@ -45,64 +45,64 @@ On your GitLab server: 1. Open the configuration file. - For Omnibus GitLab: + For Omnibus GitLab: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. 1. Add the provider configuration: - For Omnibus GitLab: + For Omnibus GitLab: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "google_oauth2", - "app_id" => "YOUR_APP_ID", - "app_secret" => "YOUR_APP_SECRET", - "args" => { "access_type" => "offline", "approval_prompt" => '' } - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "google_oauth2", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "access_type" => "offline", "approval_prompt" => '' } + } + ] + ``` - For installations from source: + For installations from source: - ```yaml - - { name: 'google_oauth2', app_id: 'YOUR_APP_ID', - app_secret: 'YOUR_APP_SECRET', - args: { access_type: 'offline', approval_prompt: '' } } - ``` + ```yaml + - { name: 'google_oauth2', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { access_type: 'offline', approval_prompt: '' } } + ``` 1. Change `YOUR_APP_ID` to the client ID from the Google Developer page 1. Similarly, change `YOUR_APP_SECRET` to the client secret 1. Make sure that you configure GitLab to use an FQDN as Google will not accept raw IP addresses. - For Omnibus packages: + For Omnibus packages: - ```ruby - external_url 'https://gitlab.example.com' - ``` + ```ruby + external_url 'https://gitlab.example.com' + ``` - For installations from source: + For installations from source: - ```yaml - gitlab: - host: https://gitlab.example.com - ``` + ```yaml + gitlab: + host: https://gitlab.example.com + ``` -1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. Save the configuration file. +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a Google icon below the regular sign in form. Click the icon to begin the authentication process. Google will ask the diff --git a/doc/integration/img/limit_namespace_filter.png b/doc/integration/img/limit_namespace_filter.png Binary files differindex 88f5caa41db..437aecad467 100644 --- a/doc/integration/img/limit_namespace_filter.png +++ b/doc/integration/img/limit_namespace_filter.png diff --git a/doc/integration/img/limit_namespaces_projects_options.png b/doc/integration/img/limit_namespaces_projects_options.png Binary files differindex 488341f7e92..fa666c7491e 100644 --- a/doc/integration/img/limit_namespaces_projects_options.png +++ b/doc/integration/img/limit_namespaces_projects_options.png diff --git a/doc/integration/img/salesforce_app_details.png b/doc/integration/img/salesforce_app_details.png Binary files differindex 00e66f07282..c7a4084102e 100644 --- a/doc/integration/img/salesforce_app_details.png +++ b/doc/integration/img/salesforce_app_details.png diff --git a/doc/integration/img/salesforce_app_secret_details.png b/doc/integration/img/salesforce_app_secret_details.png Binary files differindex fad2a4a1f97..8734a7a5cbb 100644 --- a/doc/integration/img/salesforce_app_secret_details.png +++ b/doc/integration/img/salesforce_app_secret_details.png diff --git a/doc/integration/img/salesforce_oauth_app_details.png b/doc/integration/img/salesforce_oauth_app_details.png Binary files differindex a5fb680cca6..e29c55df656 100644 --- a/doc/integration/img/salesforce_oauth_app_details.png +++ b/doc/integration/img/salesforce_oauth_app_details.png diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 50cb3d50009..d865f977799 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -35,6 +35,9 @@ and [Migrating from Jenkins to GitLab](https://www.youtube.com/watch?v=RlEVGOpYF For a real use case, read the blog post [Continuous integration: From Jenkins to GitLab using Docker](https://about.gitlab.com/2017/07/27/docker-my-precious/). +NOTE: **Moving from a traditional CI plug-in to a single application for the entire software development lifecycle can decrease hours spent on maintaining toolchains by 10% or more.** +Visit the ['GitLab vs. Jenkins' comparison page](https://about.gitlab.com/devops-tools/jenkins-vs-gitlab.html) to learn how our built-in CI compares to Jenkins. + ## Requirements - [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md index eae705c9637..3e437eb688a 100644 --- a/doc/integration/jenkins_deprecated.md +++ b/doc/integration/jenkins_deprecated.md @@ -14,12 +14,12 @@ Integration includes: Requirements: - [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) -- git clone access for Jenkins from GitLab repo (via ssh key) +- Git clone access for Jenkins from GitLab repo (via ssh key) ## Jenkins 1. Install [GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) -2. Set up jenkins project +1. Set up Jenkins project  diff --git a/doc/integration/jira.md b/doc/integration/jira.md index c09fde08326..37eba25fb5a 100644 --- a/doc/integration/jira.md +++ b/doc/integration/jira.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/jira.md' --- -This document was moved to [integrations/jira](../user/project/integrations/jira.md). +This document was moved to [another location](../user/project/integrations/jira.md). diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md index 60c7bdabf93..c413e07ec93 100644 --- a/doc/integration/jira_development_panel.md +++ b/doc/integration/jira_development_panel.md @@ -23,7 +23,7 @@ or instance admin (in the case of self-hosted GitLab) set up the integration, in order to simplify administration. TIP: **Tip:** -Create and use a single-purpose "jira" user in GitLab, so that removing +Create and use a single-purpose `jira` user in GitLab, so that removing regular users won't impact your integration. ## Requirements @@ -43,67 +43,68 @@ There are no special requirements if you are using GitLab.com. 1. In GitLab, create a new application in order to allow Jira to connect with your GitLab account - While logged-in, go to `Settings -> Applications`. (Click your profile avatar at - the top right, choose `Settings`, and then navigate to `Applications` from the left - navigation menu.) Use the form to create a new application. + While logged-in, go to `Settings -> Applications`. (Click your profile avatar at + the top right, choose `Settings`, and then navigate to `Applications` from the left + navigation menu.) Use the form to create a new application. - Enter a useful name for the `Name` field. + Enter a useful name for the `Name` field. - For the `Redirect URI` field, enter `https://<your-gitlab-instance-domain>/login/oauth/callback`, - replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, - this would be `https://gitlab.com/login/oauth/callback`. + For the `Redirect URI` field, enter `https://<your-gitlab-instance-domain>/login/oauth/callback`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/login/oauth/callback`. - NOTE: **Note**: - If using a GitLab version earlier than 11.3 the `Redirect URI` value should be `https://<your-gitlab-instance-domain>/-/jira/login/oauth/callback`. + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Redirect URI` value should be `https://<your-gitlab-instance-domain>/-/jira/login/oauth/callback`. -  - - Check `api` in the Scopes section. +  -2. Click `Save application`. You will see the generated 'Application Id' and 'Secret' values. - Copy these values that you will use on the Jira configuration side. + - Check `api` in the Scopes section. + +1. Click `Save application`. You will see the generated 'Application Id' and 'Secret' values. + Copy these values that you will use on the Jira configuration side. ## Jira Configuration 1. In Jira, from the gear menu at the top right, go to `Applications`. Navigate to `DVCS accounts` - from the left navigation menu. Click `Link GitHub account` to start creating a new integration. - (We are pretending to be GitHub in this integration until there is further platform support from Jira.) + from the left navigation menu. Click `Link GitHub account` to start creating a new integration. + (We are pretending to be GitHub in this integration until there is further platform support from Jira.) -  +  -2. Complete the form +1. Complete the form - Select GitHub Enterprise for the `Host` field. + Select GitHub Enterprise for the `Host` field. - For the `Team or User Account` field, enter the relative path of a top-level GitLab group that you have access to, - or the relative path of your personal namespace. + For the `Team or User Account` field, enter the relative path of a top-level GitLab group that you have access to, + or the relative path of your personal namespace. -  +  - For the `Host URL` field, enter `https://<your-gitlab-instance-domain>/`, - replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, - this would be `https://gitlab.com/`. + For the `Host URL` field, enter `https://<your-gitlab-instance-domain>/`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/`. - NOTE: **Note**: - If using a GitLab version earlier than 11.3 the `Host URL` value should be `https://<your-gitlab-instance-domain>/-/jira` + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Host URL` value should be `https://<your-gitlab-instance-domain>/-/jira` - For the `Client ID` field, use the `Application ID` value from the previous section. + For the `Client ID` field, use the `Application ID` value from the previous section. - For the `Client Secret` field, use the `Secret` value from the previous section. + For the `Client Secret` field, use the `Secret` value from the previous section. - Ensure that the rest of the checkboxes are checked. + Ensure that the rest of the checkboxes are checked. -3. Click `Add` to complete and create the integration. +1. Click `Add` to complete and create the integration. - Jira takes up to a few minutes to know about (import behind the scenes) all the commits and branches - for all the projects in the GitLab group you specified in the previous step. These are refreshed - every 60 minutes. + Jira takes up to a few minutes to know about (import behind the scenes) all the commits and branches + for all the projects in the GitLab group you specified in the previous step. These are refreshed + every 60 minutes. - > **Note:** - > In the future, we plan on implementating real-time integration. If you need - > to refresh the data manually, you can do this from the `Applications -> DVCS - > accounts` screen where you initially set up the integration: - > - >  + > **Note:** + > In the future, we plan on implementating real-time integration. If you need + > to refresh the data manually, you can do this from the `Applications -> DVCS + > accounts` screen where you initially set up the integration: + > + >  To connect additional GitLab projects from other GitLab top-level groups (or personal namespaces), repeat the above steps with additional Jira DVCS accounts. diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index cf2ef511264..81a1e9b0067 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -46,45 +46,41 @@ sudo chmod 0600 /etc/http.keytab For source installations, make sure the `kerberos` gem group [has been installed](../install/installation.md#install-gems). -1. Edit the kerberos section of [gitlab.yml] to enable Kerberos ticket-based +1. Edit the `kerberos` section of [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example) to enable Kerberos ticket-based authentication. In most cases, you only need to enable Kerberos and specify the location of the keytab: - ```yaml - omniauth: - enabled: true - allow_single_sign_on: ['kerberos'] + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ['kerberos'] - kerberos: - # Allow the HTTP Negotiate authentication method for Git clients - enabled: true + kerberos: + # Allow the HTTP Negotiate authentication method for Git clients + enabled: true - # Kerberos 5 keytab file. The keytab file must be readable by the GitLab user, - # and should be different from other keytabs in the system. - # (default: use default keytab from Krb5 config) - keytab: /etc/http.keytab - ``` + # Kerberos 5 keytab file. The keytab file must be readable by the GitLab user, + # and should be different from other keytabs in the system. + # (default: use default keytab from Krb5 config) + keytab: /etc/http.keytab + ``` 1. [Restart GitLab] for the changes to take effect. ---- - **Omnibus package installations** 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['omniauth_enabled'] = true - gitlab_rails['omniauth_allow_single_sign_on'] = ['kerberos'] + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = ['kerberos'] - gitlab_rails['kerberos_enabled'] = true - gitlab_rails['kerberos_keytab'] = "/etc/http.keytab" - ``` + gitlab_rails['kerberos_enabled'] = true + gitlab_rails['kerberos_keytab'] = "/etc/http.keytab" + ``` 1. [Reconfigure GitLab] for the changes to take effect. ---- - GitLab will now offer the `negotiate` authentication method for signing in and HTTP Git access, enabling Git clients that support this authentication protocol to authenticate with Kerberos tokens. @@ -149,45 +145,41 @@ keep offering only `basic` authentication. (e.g., `/etc/nginx/sites-available/gitlab-ssl`) and configure NGINX to listen to port `8443` in addition to the standard HTTPS port: - ```conf - server { - listen 0.0.0.0:443 ssl; - listen [::]:443 ipv6only=on ssl default_server; - listen 0.0.0.0:8443 ssl; - listen [::]:8443 ipv6only=on ssl; - ``` - -1. Update the Kerberos section of [gitlab.yml]: - - ```yaml - kerberos: - # Dedicated port: Git before 2.4 does not fall back to Basic authentication if Negotiate fails. - # To support both Basic and Negotiate methods with older versions of Git, configure - # nginx to proxy GitLab on an extra port (e.g. 8443) and uncomment the following lines - # to dedicate this port to Kerberos authentication. (default: false) - use_dedicated_port: true - port: 8443 - https: true - ``` + ```conf + server { + listen 0.0.0.0:443 ssl; + listen [::]:443 ipv6only=on ssl default_server; + listen 0.0.0.0:8443 ssl; + listen [::]:8443 ipv6only=on ssl; + ``` + +1. Update the `kerberos` section of [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example): + + ```yaml + kerberos: + # Dedicated port: Git before 2.4 does not fall back to Basic authentication if Negotiate fails. + # To support both Basic and Negotiate methods with older versions of Git, configure + # nginx to proxy GitLab on an extra port (e.g. 8443) and uncomment the following lines + # to dedicate this port to Kerberos authentication. (default: false) + use_dedicated_port: true + port: 8443 + https: true + ``` 1. [Restart GitLab] and NGINX for the changes to take effect. ---- - **For Omnibus package installations** 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['kerberos_use_dedicated_port'] = true - gitlab_rails['kerberos_port'] = 8443 - gitlab_rails['kerberos_https'] = true - ``` + ```ruby + gitlab_rails['kerberos_use_dedicated_port'] = true + gitlab_rails['kerberos_port'] = 8443 + gitlab_rails['kerberos_https'] = true + ``` 1. [Reconfigure GitLab] for the changes to take effect. ---- - After this change, all Git remote URLs will have to be updated to `https://gitlab.example.com:8443/mygroup/myproject.git` in order to use Kerberos ticket-based authentication. @@ -211,30 +203,28 @@ remove the OmniAuth provider named `kerberos` from your `gitlab.yml` / **For installations from source** -1. Edit [gitlab.yml] and remove the `- { name: 'kerberos' }` line under omniauth +1. Edit [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example) and remove the `- { name: 'kerberos' }` line under omniauth providers: - ```yaml - omniauth: - # ... - providers: - - { name: 'kerberos' } # <-- remove this line - ``` + ```yaml + omniauth: + # ... + providers: + - { name: 'kerberos' } # <-- remove this line + ``` 1. [Restart GitLab] for the changes to take effect. ---- - **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb` and remove the `{ "name" => "kerberos" }` line under `gitlab_rails['omniauth_providers']`: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { "name" => "kerberos" } # <-- remove this entry - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { "name" => "kerberos" } # <-- remove this entry + ] + ``` 1. [Reconfigure GitLab] for the changes to take effect. @@ -290,7 +280,7 @@ remote: HTTP Basic: Access denied fatal: Authentication failed for '<KRB5 path>' ``` -If you are using Git v2.11 or newer and see the above error when cloning, you can +If you are using Git v2.11 or newer and see the above error when cloning, you can set the `http.emptyAuth` Git option to `true` to fix this: ``` @@ -305,7 +295,6 @@ See also: [Git v2.11 release notes](https://github.com/git/git/blob/master/Docum - <http://blog.manula.org/2012/04/setting-up-kerberos-server-with-debian.html> - <http://www.roguelynn.com/words/explain-like-im-5-kerberos/> -[gitlab.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example [restart gitlab]: ../administration/restart_gitlab.md#installations-from-source [reconfigure gitlab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure [nginx]: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md index 3c1a0f2a117..ad59592c45f 100644 --- a/doc/integration/oauth2_generic.md +++ b/doc/integration/oauth2_generic.md @@ -12,7 +12,7 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc 1. Strategy parses user information from the response, using a **configurable** format 1. GitLab finds or creates the returned user and logs them in -## Limitations of this Strategy: +## Limitations of this Strategy - It can only be used for Single Sign on, and will not provide any other access granted by any OAuth provider (importing projects or users, etc) @@ -24,11 +24,11 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc 1. Register your application in the OAuth2 provider you wish to authenticate with. - The redirect URI you provide when registering the application should be: + The redirect URI you provide when registering the application should be: - ``` - http://your-gitlab.host.com/users/auth/oauth2_generic/callback - ``` + ``` + http://your-gitlab.host.com/users/auth/oauth2_generic/callback + ``` 1. You should now be able to get a Client ID and Client Secret. Where this shows up will differ for each provider. @@ -36,18 +36,18 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc 1. On your GitLab server, open the configuration file. - For Omnibus package: + For Omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` 1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index b9dc2e123c5..36b4836e6b3 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -25,8 +25,6 @@ can be used for authentication to your GitLab instance The 'GitLab Importer' feature is also using the OAuth protocol to give access to repositories without sharing user credentials to your GitLab.com account. ---- - GitLab supports two ways of adding a new OAuth2 application to an instance. You can either add an application as a regular user or add it in the admin area. What this means is that GitLab can actually have instance-wide and a user-wide @@ -41,24 +39,18 @@ In order to add a new application via your profile, navigate to  ---- - In the application form, enter a **Name** (arbitrary), and make sure to set up correctly the **Redirect URI** which is the URL where users will be sent after they authorize with GitLab.  ---- - When you hit **Submit** you will be provided with the application ID and the application secret which you can then use with your application that connects to GitLab.  ---- - ## OAuth applications in the admin area To create an application that does not belong to a certain user, you can create @@ -69,8 +61,6 @@ it from the admin area. You're also able to mark an application as _trusted_ when creating it through the admin area. By doing that, the user authorization step is automatically skipped for this application. ---- - ## Authorized applications Every application you authorized to use your GitLab credentials will be shown @@ -78,8 +68,6 @@ in the **Authorized applications** section under **Profile Settings > Applicatio  ---- - GitLab's OAuth applications support scopes, which allow various actions that any given application can perform such as `read_user` and `api`. There are many more scopes available. diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index bf5debc7694..ef319f7f0ce 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -69,59 +69,59 @@ that are in common for all providers that we need to consider. To change these settings: -- **For omnibus package** +- **For Omnibus package** - Open the configuration file: + Open the configuration file: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - and change: + and change: - ```ruby - # Versions prior to 11.4 require this to be set to true - # gitlab_rails['omniauth_enabled'] = nil + ```ruby + # Versions prior to 11.4 require this to be set to true + # gitlab_rails['omniauth_enabled'] = nil - # CAUTION! - # This allows users to login without having a user account first. Define the allowed providers - # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. - # User accounts will be created automatically when authentication was successful. - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml', 'twitter'] - gitlab_rails['omniauth_auto_link_ldap_user'] = true - gitlab_rails['omniauth_block_auto_created_users'] = true - ``` + # CAUTION! + # This allows users to login without having a user account first. Define the allowed providers + # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml', 'twitter'] + gitlab_rails['omniauth_auto_link_ldap_user'] = true + gitlab_rails['omniauth_block_auto_created_users'] = true + ``` - **For installations from source** - Open the configuration file: + Open the configuration file: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` - and change the following section: + and change the following section: - ```yaml - ## OmniAuth settings - omniauth: - # Allow login via Twitter, Google, etc. using OmniAuth providers - # Versions prior to 11.4 require this to be set to true - # enabled: true + ```yaml + ## OmniAuth settings + omniauth: + # Allow login via Twitter, Google, etc. using OmniAuth providers + # Versions prior to 11.4 require this to be set to true + # enabled: true - # CAUTION! - # This allows users to login without having a user account first. Define the allowed providers - # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. - # User accounts will be created automatically when authentication was successful. - allow_single_sign_on: ["saml", "twitter"] + # CAUTION! + # This allows users to login without having a user account first. Define the allowed providers + # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + allow_single_sign_on: ["saml", "twitter"] - auto_link_ldap_user: true + auto_link_ldap_user: true - # Locks down those users until they have been cleared by the admin (default: true). - block_auto_created_users: true - ``` + # Locks down those users until they have been cleared by the admin (default: true). + block_auto_created_users: true + ``` Now we can choose one or more of the [Supported Providers](#supported-providers) listed above to continue the configuration process. @@ -161,14 +161,14 @@ want their accounts to be upgraded to full internal accounts. **For Omnibus installations** ```ruby - gitlab_rails['omniauth_external_providers'] = ['twitter', 'google_oauth2'] +gitlab_rails['omniauth_external_providers'] = ['twitter', 'google_oauth2'] ``` **For installations from source** ```yaml - omniauth: - external_providers: ['twitter', 'google_oauth2'] +omniauth: + external_providers: ['twitter', 'google_oauth2'] ``` ## Using Custom Omniauth Providers @@ -186,23 +186,31 @@ these cases you can use the Omniauth provider. These steps are fairly general and you will need to figure out the exact details from the Omniauth provider's documentation. -- Stop GitLab: +- Stop GitLab: - sudo service gitlab stop + ```sh + sudo service gitlab stop + ``` -- Add the gem to your [Gemfile](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile): +- Add the gem to your [Gemfile](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Gemfile): - gem "omniauth-your-auth-provider" + ```sh + gem "omniauth-your-auth-provider" + ``` -- Install the new Omniauth provider gem by running the following command: +- Install the new Omniauth provider gem by running the following command: - sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment + ```sh + sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment + ``` - > These are the same commands you used during initial installation in the [Install Gems section](../install/installation.md#install-gems) with `--path vendor/bundle --no-deployment` instead of `--deployment`. + > These are the same commands you used during initial installation in the [Install Gems section](../install/installation.md#install-gems) with `--path vendor/bundle --no-deployment` instead of `--deployment`. -- Start GitLab: +- Start GitLab: - sudo service gitlab start + ```sh + sudo service gitlab start + ``` ### Examples @@ -247,8 +255,8 @@ gitlab_rails['omniauth_enabled'] = false **For installations from source** ```yaml - omniauth: - enabled: false +omniauth: + enabled: false ``` ## Keep OmniAuth user profiles up to date @@ -258,14 +266,35 @@ You can enable profile syncing from selected OmniAuth providers and for all or f When authenticating using LDAP, the user's name and email are always synced. ```ruby - gitlab_rails['sync_profile_from_provider'] = ['twitter', 'google_oauth2'] - gitlab_rails['sync_profile_attributes'] = ['name', 'email', 'location'] - ``` +gitlab_rails['sync_profile_from_provider'] = ['twitter', 'google_oauth2'] +gitlab_rails['sync_profile_attributes'] = ['name', 'email', 'location'] +``` + +**For installations from source** + +```yaml +omniauth: + sync_profile_from_provider: ['twitter', 'google_oauth2'] + sync_profile_attributes: ['email', 'location'] +``` + +## Bypassing two factor authentication + +Starting with GitLab 12.3, this allows users to login with the specified +providers without two factor authentication. + +Define the allowed providers using an array, e.g. `["twitter", 'google_oauth2']`, or as +`true`/`false` to allow all providers or none. This option should only be configured +for providers which already have two factor authentication (default: false). +This configration dose not apply to SAML. + +```ruby +gitlab_rails['omniauth_allow_bypass_two_factor'] = ['twitter', 'google_oauth2'] +``` **For installations from source** ```yaml - omniauth: - sync_profile_from_provider: ['twitter', 'google_oauth2'] - sync_profile_attributes: ['email', 'location'] +omniauth: + allow_bypass_two_factor: ['twitter', 'google_oauth2'] ``` diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index 1ef43cfcece..10ab9d3c126 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -7,73 +7,77 @@ You can integrate your GitLab instance with [Salesforce](https://www.salesforce. To enable Salesforce OmniAuth provider, you must use Salesforce's credentials for your GitLab instance. To get the credentials (a pair of Client ID and Client Secret), you must [create a Connected App](https://help.salesforce.com/articleView?id=connected_app_create.htm&type=5) on Salesforce. -1. Sign in to [Salesforce](https://login.salesforce.com/). +1. Sign in to [Salesforce](https://login.salesforce.com/). -1. In Setup, enter `App Manager` in the Quick Find box, click **App Manager**, then click **New Connected App**. +1. In Setup, enter `App Manager` in the Quick Find box, click **App Manager**, then click **New Connected App**. -1. Fill in the application details into the following fields: - - **Connected App Name** and **API Name**: Set to any value but consider something like `<Organization>'s GitLab`, `<Your Name>'s GitLab`, or something else that is descriptive. - - **Contact Email**: Enter the contact email for Salesforce to use when contacting you or your support team. - - **Description**: Description for the application. +1. Fill in the application details into the following fields: + - **Connected App Name** and **API Name**: Set to any value but consider something like `<Organization>'s GitLab`, `<Your Name>'s GitLab`, or something else that is descriptive. + - **Contact Email**: Enter the contact email for Salesforce to use when contacting you or your support team. + - **Description**: Description for the application. -  -1. Select **API (Enable OAuth Settings)** and click on **Enable OAuth Settings**. -1. Fill in the application details into the following fields: - - **Callback URL**: The callback URL of your GitLab installation. For example, `https://gitlab.example.com/users/auth/salesforce/callback`. - - **Selected OAuth Scopes**: Move **Access your basic information (id, profile, email, address, phone)** and **Allow access to your unique identifier (openid)** to the right column. +  + +1. Select **API (Enable OAuth Settings)** and click on **Enable OAuth Settings**. +1. Fill in the application details into the following fields: + - **Callback URL**: The callback URL of your GitLab installation. For example, `https://gitlab.example.com/users/auth/salesforce/callback`. + - **Selected OAuth Scopes**: Move **Access your basic information (id, profile, email, address, phone)** and **Allow access to your unique identifier (openid)** to the right column. + +  -  1. Click **Save**. -1. On your GitLab server, open the configuration file. +1. On your GitLab server, open the configuration file. + + For Omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For omnibus package: + For installations from source: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` - For installations from source: +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` +1. Add the provider configuration: -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + For Omnibus package: -1. Add the provider configuration: + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "salesforce", + "app_id" => "SALESFORCE_CLIENT_ID", + "app_secret" => "SALESFORCE_CLIENT_SECRET" + } + ] + ``` - For omnibus package: + For installation from source: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "salesforce", - "app_id" => "SALESFORCE_CLIENT_ID", - "app_secret" => "SALESFORCE_CLIENT_SECRET" - } - ] - ``` + ``` + - { name: 'salesforce', + app_id: 'SALESFORCE_CLIENT_ID', + app_secret: 'SALESFORCE_CLIENT_SECRET' + } + ``` - For installation from source: +1. Change `SALESFORCE_CLIENT_ID` to the Consumer Key from the Salesforce connected application page. +1. Change `SALESFORCE_CLIENT_SECRET` to the Consumer Secret from the Salesforce connected application page. - ``` - - { name: 'salesforce', - app_id: 'SALESFORCE_CLIENT_ID', - app_secret: 'SALESFORCE_CLIENT_SECRET' - } - ``` -1. Change `SALESFORCE_CLIENT_ID` to the Consumer Key from the Salesforce connected application page. -1. Change `SALESFORCE_CLIENT_SECRET` to the Consumer Secret from the Salesforce connected application page. -  +  -1. Save the configuration file. -1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. +1. Save the configuration file. +1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be a Salesforce icon below the regular sign in form. Click the icon to begin the authentication process. Salesforce will ask the user to sign in and authorize the GitLab application. If everything goes well, the user will be returned to GitLab and will be signed in. NOTE: **Note:** -GitLab requires the email address of each new user. Once the user is logged in using Salesforce, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email.
\ No newline at end of file +GitLab requires the email address of each new user. Once the user is logged in using Salesforce, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 22e07594d6f..de160e72dda 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -17,7 +17,7 @@ in your SAML IdP: 1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: ```sh sudo editor /etc/gitlab/gitlab.rb @@ -34,7 +34,7 @@ in your SAML IdP: 1. To allow your users to use SAML to sign up without having to manually create an account first, don't forget to add the following values to your configuration: - For omnibus package: + For Omnibus package: ```ruby gitlab_rails['omniauth_enabled'] = true @@ -54,7 +54,7 @@ in your SAML IdP: 1. You can also automatically link SAML users with existing GitLab users if their email addresses match by adding the following setting: - For omnibus package: + For Omnibus package: ```ruby gitlab_rails['omniauth_auto_link_saml_user'] = true @@ -68,7 +68,7 @@ in your SAML IdP: 1. Add the provider configuration: - For omnibus package: + For Omnibus package: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -342,7 +342,7 @@ You can add this setting to your GitLab configuration to automatically redirect to your SAML server for authentication, thus removing the need to click a button before actually signing in. -For omnibus package: +For Omnibus package: ```ruby gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml' diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index 07c83c1a049..ca5a8077e73 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -1,48 +1,48 @@ # Shibboleth OmniAuth Provider -This documentation is for enabling shibboleth with omnibus-gitlab package. +This documentation is for enabling Shibboleth with the Omnibus GitLab package. -In order to enable Shibboleth support in gitlab we need to use Apache instead of Nginx (It may be possible to use Nginx, however this is difficult to configure using the bundled Nginx provided in the omnibus-gitlab package). Apache uses mod_shib2 module for shibboleth authentication and can pass attributes as headers to omniauth-shibboleth provider. +In order to enable Shibboleth support in GitLab we need to use Apache instead of Nginx (It may be possible to use Nginx, however this is difficult to configure using the bundled Nginx provided in the Omnibus GitLab package). Apache uses mod_shib2 module for Shibboleth authentication and can pass attributes as headers to Omniauth Shibboleth provider. -To enable the Shibboleth OmniAuth provider you must configure Apache shibboleth module. +To enable the Shibboleth OmniAuth provider you must configure Apache Shibboleth module. The installation and configuration of the module itself is out of the scope of this document. Check <https://wiki.shibboleth.net/confluence/display/SP3/Apache> for more info. -You can find Apache config in gitlab-recipes (<https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache>). +You can find Apache config in [GitLab Recipes](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache). The following changes are needed to enable Shibboleth: -1. Protect omniauth-shibboleth callback URL: - - ``` - <Location /users/auth/shibboleth/callback> - AuthType shibboleth - ShibRequestSetting requireSession 1 - ShibUseHeaders On - require valid-user - </Location> - - Alias /shibboleth-sp /usr/share/shibboleth - <Location /shibboleth-sp> - Satisfy any - </Location> - - <Location /Shibboleth.sso> - SetHandler shib - </Location> - ``` - -1. Exclude shibboleth URLs from rewriting. Add `RewriteCond %{REQUEST_URI} !/Shibboleth.sso` and `RewriteCond %{REQUEST_URI} !/shibboleth-sp`. Config should look like this: - - ``` - # Apache equivalent of Nginx try files - RewriteEngine on - RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f - RewriteCond %{REQUEST_URI} !/Shibboleth.sso - RewriteCond %{REQUEST_URI} !/shibboleth-sp - RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] - RequestHeader set X_FORWARDED_PROTO 'https' - ``` +1. Protect Omniauth Shibboleth callback URL: + + ``` + <Location /users/auth/shibboleth/callback> + AuthType shibboleth + ShibRequestSetting requireSession 1 + ShibUseHeaders On + require valid-user + </Location> + + Alias /shibboleth-sp /usr/share/shibboleth + <Location /shibboleth-sp> + Satisfy any + </Location> + + <Location /Shibboleth.sso> + SetHandler shib + </Location> + ``` + +1. Exclude Shibboleth URLs from rewriting. Add `RewriteCond %{REQUEST_URI} !/Shibboleth.sso` and `RewriteCond %{REQUEST_URI} !/shibboleth-sp`. Config should look like this: + + ``` + # Apache equivalent of Nginx try files + RewriteEngine on + RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteRule .* http://127.0.0.1:8080%{REQUEST_URI} [P,QSA] + RequestHeader set X_FORWARDED_PROTO 'https' + ``` 1. Edit `/etc/gitlab/gitlab.rb` configuration file to enable OmniAuth and add Shibboleth as an OmniAuth provider. User attributes will be sent from the @@ -50,7 +50,7 @@ The following changes are needed to enable Shibboleth: attribute mapping. Therefore the values of the `args` hash should be in the form of `"HTTP_ATTRIBUTE"`. The keys in the hash are arguments to the [OmniAuth::Strategies::Shibboleth class](https://github.com/toyokazu/omniauth-shibboleth/blob/master/lib/omniauth/strategies/shibboleth.rb) - and are documented by the [omniauth-shibboleth gem](https://github.com/toyokazu/omniauth-shibboleth) + and are documented by the [`omniauth-shibboleth` gem](https://github.com/toyokazu/omniauth-shibboleth) (take care to note the version of the gem packaged with GitLab). If some of your users appear to be authenticated by Shibboleth and Apache, but GitLab rejects their account with a URI that contains "e-mail is invalid" then your @@ -60,31 +60,31 @@ The following changes are needed to enable Shibboleth: The file should look like this: - ``` - external_url 'https://gitlab.example.com' - gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - - # disable Nginx - nginx['enable'] = false - - gitlab_rails['omniauth_allow_single_sign_on'] = true - gitlab_rails['omniauth_block_auto_created_users'] = false - gitlab_rails['omniauth_enabled'] = true - gitlab_rails['omniauth_providers'] = [ - { - "name" => "'shibboleth"', - "label" => "Text for Login Button", - "args" => { - "shib_session_id_field" => "HTTP_SHIB_SESSION_ID", - "shib_application_id_field" => "HTTP_SHIB_APPLICATION_ID", - "uid_field" => 'HTTP_EPPN', - "name_field" => 'HTTP_CN', - "info_fields" => { "email" => 'HTTP_MAIL'} - } - } - ] - - ``` + ``` + external_url 'https://gitlab.example.com' + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # disable Nginx + nginx['enable'] = false + + gitlab_rails['omniauth_allow_single_sign_on'] = true + gitlab_rails['omniauth_block_auto_created_users'] = false + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_providers'] = [ + { + "name" => "'shibboleth"', + "label" => "Text for Login Button", + "args" => { + "shib_session_id_field" => "HTTP_SHIB_SESSION_ID", + "shib_application_id_field" => "HTTP_SHIB_APPLICATION_ID", + "uid_field" => 'HTTP_EPPN', + "name_field" => 'HTTP_CN', + "info_fields" => { "email" => 'HTTP_MAIL'} + } + } + ] + + ``` 1. [Reconfigure](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../administration/restart_gitlab.md#installations-from-source) GitLab for the changes to take effect if you installed GitLab via Omnibus or from source respectively. @@ -94,47 +94,47 @@ On the sign in page, there should now be a "Sign in with: Shibboleth" icon below ## Apache 2.4 / GitLab 8.6 update The order of the first 2 Location directives is important. If they are reversed, -you will not get a shibboleth session! +you will not get a Shibboleth session! ``` - <Location /> - Require all granted - ProxyPassReverse http://127.0.0.1:8181 - ProxyPassReverse http://YOUR_SERVER_FQDN/ - </Location> - - <Location /users/auth/shibboleth/callback> - AuthType shibboleth - ShibRequestSetting requireSession 1 - ShibUseHeaders On - Require shib-session - </Location> - - Alias /shibboleth-sp /usr/share/shibboleth - - <Location /shibboleth-sp> - Require all granted - </Location> - - <Location /Shibboleth.sso> - SetHandler shib - </Location> - - RewriteEngine on - - #Don't escape encoded characters in api requests - RewriteCond %{REQUEST_URI} ^/api/v4/.* - RewriteCond %{REQUEST_URI} !/Shibboleth.sso - RewriteCond %{REQUEST_URI} !/shibboleth-sp - RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE] - - #Forward all requests to gitlab-workhorse except existing files - RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR] - RewriteCond %{REQUEST_URI} ^/uploads/.* - RewriteCond %{REQUEST_URI} !/Shibboleth.sso - RewriteCond %{REQUEST_URI} !/shibboleth-sp - RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] - - RequestHeader set X_FORWARDED_PROTO 'https' - RequestHeader set X-Forwarded-Ssl on +<Location /> + Require all granted + ProxyPassReverse http://127.0.0.1:8181 + ProxyPassReverse http://YOUR_SERVER_FQDN/ +</Location> + +<Location /users/auth/shibboleth/callback> + AuthType shibboleth + ShibRequestSetting requireSession 1 + ShibUseHeaders On + Require shib-session +</Location> + +Alias /shibboleth-sp /usr/share/shibboleth + +<Location /shibboleth-sp> + Require all granted +</Location> + +<Location /Shibboleth.sso> + SetHandler shib +</Location> + +RewriteEngine on + +#Don't escape encoded characters in api requests +RewriteCond %{REQUEST_URI} ^/api/v4/.* +RewriteCond %{REQUEST_URI} !/Shibboleth.sso +RewriteCond %{REQUEST_URI} !/shibboleth-sp +RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE] + +#Forward all requests to gitlab-workhorse except existing files +RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR] +RewriteCond %{REQUEST_URI} ^/uploads/.* +RewriteCond %{REQUEST_URI} !/Shibboleth.sso +RewriteCond %{REQUEST_URI} !/shibboleth-sp +RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] + +RequestHeader set X_FORWARDED_PROTO 'https' +RequestHeader set X-Forwarded-Ssl on ``` diff --git a/doc/integration/slack.md b/doc/integration/slack.md index f84ab769218..9fcf2c2d99a 100644 --- a/doc/integration/slack.md +++ b/doc/integration/slack.md @@ -1,5 +1,5 @@ --- -redirect_to: '../project_services/slack.md' +redirect_to: '../user/project/integrations/slack.md' --- -This document was moved to [project_services/slack.md](../project_services/slack.md). +This document was moved to [project_services/slack.md](../user/project/integrations/slack.md). diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index d8096993885..b2bd1b57d0f 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -32,7 +32,7 @@ To enable the Twitter OmniAuth provider you must register your application with 1. On your GitLab server, open the configuration file. - For omnibus package: + For Omnibus package: ```sh sudo editor /etc/gitlab/gitlab.rb @@ -50,7 +50,7 @@ To enable the Twitter OmniAuth provider you must register your application with 1. Add the provider configuration: - For omnibus package: + For Omnibus package: ```ruby gitlab_rails['omniauth_providers'] = [ diff --git a/doc/integration/ultra_auth.md b/doc/integration/ultra_auth.md index 69b2a75050d..fb950ba989a 100644 --- a/doc/integration/ultra_auth.md +++ b/doc/integration/ultra_auth.md @@ -7,69 +7,78 @@ You can integrate your GitLab instance with [UltraAuth](https://ultraauth.com) t To enable UltraAuth OmniAuth provider, you must use UltraAuth's credentials for your GitLab instance. To get the credentials (a pair of Client ID and Client Secret), you must register an application on UltraAuth. -1. Sign in to [UltraAuth](https://ultraauth.com). -1. Navigate to [Create an App](https://ultraauth.com/select-strategy) and click on "Ruby on Rails". -1. Scroll down the page that is displayed to locate the **Client ID** and **Client Secret**. - Keep this page open as you continue configuration. -  -1. Click on "Edit Callback URL" link. -  -1. The callback URL will be `http(s)://<your_domain>/users/auth/ultraauth/callback` -  -1. Select **Register application**. -1. On your GitLab server, open the configuration file. - - For omnibus package: - - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` - - For installations from source: - - ```sh - cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. -1. Add the provider configuration: - - For omnibus package: - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - "name" => "ultraauth", - "app_id" => "OPENID_CLIENT_ID", - "app_secret" => "OPENID_CLIENT_SECRET", - "args" => { - "client_options" => { - "redirect_uri" => "https://example.com/users/auth/ultraauth/callback" - } - } - } - ] - ``` - - For installation from source: - - ``` - - { name: 'ultraauth', - app_id: 'OPENID_CLIENT_ID', - app_secret: 'OPENID_CLIENT_SECRET', - args: { - client_options: { - redirect_uri: 'https://example.com/users/auth/ultraauth/callback' - } - } - } - ``` - __Replace `https://example.com/users/auth/ultraauth/callback` with your application's Callback URL.__ -1. Change `OPENID_CLIENT_ID` to the Client ID from the UltraAuth application page. -1. Change `OPENID_CLIENT_SECRET` to the Client Secret from the UltraAuth application page. -1. Save the configuration file. -1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you - installed GitLab via Omnibus or from source respectively. +1. Sign in to [UltraAuth](https://ultraauth.com). +1. Navigate to [Create an App](https://ultraauth.com/select-strategy) and click on "Ruby on Rails". +1. Scroll down the page that is displayed to locate the **Client ID** and **Client Secret**. + Keep this page open as you continue configuration. + +  + +1. Click on "Edit Callback URL" link. + +  + +1. The callback URL will be `http(s)://<your_domain>/users/auth/ultraauth/callback` + +  + +1. Select **Register application**. +1. On your GitLab server, open the configuration file. + + For Omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. Add the provider configuration: + + For Omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "ultraauth", + "app_id" => "OPENID_CLIENT_ID", + "app_secret" => "OPENID_CLIENT_SECRET", + "args" => { + "client_options" => { + "redirect_uri" => "https://example.com/users/auth/ultraauth/callback" + } + } + } + ] + ``` + + For installation from source: + + ``` + - { name: 'ultraauth', + app_id: 'OPENID_CLIENT_ID', + app_secret: 'OPENID_CLIENT_SECRET', + args: { + client_options: { + redirect_uri: 'https://example.com/users/auth/ultraauth/callback' + } + } + } + ``` + + __Replace `https://example.com/users/auth/ultraauth/callback` with your application's Callback URL.__ + +1. Change `OPENID_CLIENT_ID` to the Client ID from the UltraAuth application page. +1. Change `OPENID_CLIENT_SECRET` to the Client Secret from the UltraAuth application page. +1. Save the configuration file. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be an UltraAuth icon below the regular sign in form. Click the icon to begin the authentication process. UltraAuth will ask the user to sign in and authorize the GitLab application. diff --git a/doc/legal/corporate_contributor_license_agreement.md b/doc/legal/corporate_contributor_license_agreement.md index 7f08188bd65..8c1f4a126b1 100644 --- a/doc/legal/corporate_contributor_license_agreement.md +++ b/doc/legal/corporate_contributor_license_agreement.md @@ -2,28 +2,28 @@ You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. -1. Definitions. +- **Definitions:** - "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. - "Contribution" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, that is submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + "Contribution" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, that is submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." -2. Grant of Copyright License. +- **Grant of Copyright License:** -Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. -3. Grant of Patent License. +- **Grant of Patent License:** -Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. -4. You represent that You are legally entitled to grant the above license. You represent further that each of Your employees is authorized to submit Contributions on Your behalf, but excluding employees that are designated in writing by You as "Not authorized to submit Contributions on behalf of [name of Your corporation here]." Such designations of exclusion for unauthorized employees are to be submitted via email to legal@gitlab.com. + You represent that You are legally entitled to grant the above license. You represent further that each of Your employees is authorized to submit Contributions on Your behalf, but excluding employees that are designated in writing by You as "Not authorized to submit Contributions on behalf of (name of Your corporation here)." Such designations of exclusion for unauthorized employees are to be submitted via email to legal@gitlab.com. It is Your responsibility to notify GitLab B.V. when any change is required to the list of designated employees excluded from submitting Contributions on Your behalf. Such notification should also be sent via email to legal@gitlab.com. -5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). +- **Contributions:** -6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + You represent that each of Your Contributions is Your original creation. + + Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: (named here)". -7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]". - -8. It is Your responsibility to notify GitLab.com when any change is required to the list of designated employees excluded from submitting Contributions on Your behalf per Section 4. Such notification should be sent via email to legal@gitlab.com. + You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/legal/individual_contributor_license_agreement.md b/doc/legal/individual_contributor_license_agreement.md index 59803aea080..7a071414629 100644 --- a/doc/legal/individual_contributor_license_agreement.md +++ b/doc/legal/individual_contributor_license_agreement.md @@ -2,24 +2,30 @@ You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. -1. Definitions. +- **Definitions:** - "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. - "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." -2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. +- **Grant of Copyright License:** -3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. -4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to GitLab B.V., or that your employer has executed a separate Corporate CLA with GitLab B.V.. +- **Grant of Patent License:** -5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions. + Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. -6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to GitLab B.V., or that your employer has executed a separate Corporate CLA with GitLab B.V.. -7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [insert_name_here]". +- **Contributions:** -8. You agree to notify GitLab B.V. of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. + You represent that each of Your Contributions is Your original creation. You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions. + + Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: (insert_name_here)". + + You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +You agree to notify GitLab B.V. of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 7659b743311..2b8379141c3 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -67,7 +67,7 @@ Also check on your GitLab server. ``` # On your GitLab server: # Omnibus -sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads +sudo gitlab-backup create SKIP=repositories,uploads # Source cd /home/git/gitlab @@ -181,7 +181,7 @@ sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production ### 2. SQL data and build traces Create your final CI data export. If you are converting from MySQL to - PostgreSQL, add ` MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When +PostgreSQL, add `MYSQL_TO_POSTGRESQL=1` to the end of the rake command. When the command finishes it will print the path to your data export archive; you will need this file later. @@ -227,7 +227,7 @@ sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ ``` -### 5. Import the CI data into GitLab. +### 5. Import the CI data into GitLab This step will delete any existing CI data on your GitLab server. There should be no CI data yet because you turned CI on the GitLab server off earlier. @@ -323,11 +323,15 @@ You should also make sure that you can: ### 2. Check Nginx configuration - sudo nginx -t +```sh +sudo nginx -t +``` ### 3. Restart Nginx - sudo /etc/init.d/nginx restart +```sh +sudo /etc/init.d/nginx restart +``` ### Restore from backup @@ -352,11 +356,13 @@ The fix for this is to update to Omnibus 7.14 first and then update it to 8.0. ### Permission denied when accessing /var/opt/gitlab/gitlab-ci/builds To fix that issue you have to change builds/ folder permission before doing final backup: + ``` sudo chown -R gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds ``` Then before executing `ci:migrate` you need to fix builds folder permission: + ``` sudo chown git:git /var/opt/gitlab/gitlab-ci/builds ``` @@ -365,7 +371,7 @@ sudo chown git:git /var/opt/gitlab/gitlab-ci/builds If you were migrating CI database from MySQL to PostgreSQL manually you can see errors during import about missing sequences: -```sql +```sql ALTER SEQUENCE ERROR: relation "ci_builds_id_seq" does not exist ERROR: relation "ci_commits_id_seq" does not exist diff --git a/doc/pages/getting_started_part_three.md b/doc/pages/getting_started_part_three.md index b65247ff7b7..31a01a6c83b 100644 --- a/doc/pages/getting_started_part_three.md +++ b/doc/pages/getting_started_part_three.md @@ -1,5 +1,5 @@ --- -redirect_to: '../user/project/pages/getting_started_part_three.md' +redirect_to: '../user/project/pages/custom_domains_ssl_tls_certification/index.md' --- -This document was moved to [another location](../user/project/pages/getting_started_part_three.md). +This document was moved to [another location](../user/project/pages/custom_domains_ssl_tls_certification/index.md). diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 018c273c51a..f7ba7c16a9e 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -86,6 +86,7 @@ Please see the table below for some examples: | 9.4.5 | 8.13.4 | `8.13.4` -> `8.17.7` -> `9.4.5` | `8.17.7` is the last version in version `8` | | 10.1.4 | 8.13.4 | `8.13.4 -> 8.17.7 -> 9.5.10 -> 10.1.4` | `8.17.7` is the last version in version `8`, `9.5.10` is the last version in version `9` | | 11.3.4 | 8.13.4 | `8.13.4` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.3.4` | `8.17.7` is the last version in version `8`, `9.5.10` is the last version in version `9`, `10.8.7` is the last version in version `10` | +| 12.0.2 | 11.3.4 | `11.3.4` -> `11.11.x` -> `12.0.2` | `11.11.x` is the last version in version `11` More information about the release procedures can be found in our [release documentation](https://gitlab.com/gitlab-org/release/docs). You may also want to read our diff --git a/doc/project_services/bamboo.md b/doc/project_services/bamboo.md index a1ff8909f87..b1d37898516 100644 --- a/doc/project_services/bamboo.md +++ b/doc/project_services/bamboo.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/bamboo.md' --- -This document was moved to [user/project/integrations/bamboo.md](../user/project/integrations/bamboo.md). +This document was moved to [another location](../user/project/integrations/bamboo.md). diff --git a/doc/project_services/bugzilla.md b/doc/project_services/bugzilla.md index 1abf1b28939..17dff538c0e 100644 --- a/doc/project_services/bugzilla.md +++ b/doc/project_services/bugzilla.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/bugzilla.md' --- -This document was moved to [user/project/integrations/bugzilla.md](../user/project/integrations/bugzilla.md). +This document was moved to [another location](../user/project/integrations/bugzilla.md). diff --git a/doc/project_services/emails_on_push.md b/doc/project_services/emails_on_push.md index c5ab8aa5c70..a7d91934ce9 100644 --- a/doc/project_services/emails_on_push.md +++ b/doc/project_services/emails_on_push.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/emails_on_push.md' --- -This document was moved to [user/project/integrations/emails_on_push.md](../user/project/integrations/emails_on_push.md). +This document was moved to [another location](../user/project/integrations/emails_on_push.md). diff --git a/doc/project_services/hipchat.md b/doc/project_services/hipchat.md index 4ae9f6c6b2e..a2fbbd5cce5 100644 --- a/doc/project_services/hipchat.md +++ b/doc/project_services/hipchat.md @@ -1 +1,5 @@ -This document was moved to [user/project/integrations/hipchat.md](../user/project/integrations/hipchat.md). +--- +redirect_to: '../user/project/integrations/hipchat.md' +--- + +This document was moved to [another location](../user/project/integrations/hipchat.md). diff --git a/doc/project_services/irker.md b/doc/project_services/irker.md index af47abab117..70e46b1b364 100644 --- a/doc/project_services/irker.md +++ b/doc/project_services/irker.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/irker.md' --- -This document was moved to [user/project/integrations/irker.md](../user/project/integrations/irker.md). +This document was moved to [another location](../user/project/integrations/irker.md). diff --git a/doc/project_services/jira.md b/doc/project_services/jira.md index 6a0108f400f..37eba25fb5a 100644 --- a/doc/project_services/jira.md +++ b/doc/project_services/jira.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/jira.md' --- -This document was moved to [user/project/integrations/jira.md](../user/project/integrations/jira.md). +This document was moved to [another location](../user/project/integrations/jira.md). diff --git a/doc/project_services/kubernetes.md b/doc/project_services/kubernetes.md index cfe36fcd1b2..585c5ddb002 100644 --- a/doc/project_services/kubernetes.md +++ b/doc/project_services/kubernetes.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/kubernetes.md' --- -This document was moved to [user/project/integrations/kubernetes.md](../user/project/integrations/kubernetes.md). +This document was moved to [another location](../user/project/integrations/kubernetes.md). diff --git a/doc/project_services/mattermost.md b/doc/project_services/mattermost.md index de9f4d14cf7..78888395031 100644 --- a/doc/project_services/mattermost.md +++ b/doc/project_services/mattermost.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/mattermost.md' --- -This document was moved to [user/project/integrations/mattermost.md](../user/project/integrations/mattermost.md). +This document was moved to [another location](../user/project/integrations/mattermost.md). diff --git a/doc/project_services/mattermost_slash_commands.md b/doc/project_services/mattermost_slash_commands.md index 82ec34739c1..0c2774d95e0 100644 --- a/doc/project_services/mattermost_slash_commands.md +++ b/doc/project_services/mattermost_slash_commands.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/mattermost_slash_commands.md' --- -This document was moved to [user/project/integrations/mattermost_slash_commands.md](../user/project/integrations/mattermost_slash_commands.md). +This document was moved to [another location](../user/project/integrations/mattermost_slash_commands.md). diff --git a/doc/project_services/project_services.md b/doc/project_services/project_services.md index a355851a273..2ae7a0ce3ce 100644 --- a/doc/project_services/project_services.md +++ b/doc/project_services/project_services.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/project_services.md' --- -This document was moved to [user/project/integrations/project_services.md](../user/project/integrations/project_services.md). +This document was moved to [another location](../user/project/integrations/project_services.md). diff --git a/doc/project_services/redmine.md b/doc/project_services/redmine.md index 05f28f00adc..141c72d6b6b 100644 --- a/doc/project_services/redmine.md +++ b/doc/project_services/redmine.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/redmine.md' --- -This document was moved to [user/project/integrations/redmine.md](../user/project/integrations/redmine.md). +This document was moved to [another location](../user/project/integrations/redmine.md). diff --git a/doc/project_services/services_templates.md b/doc/project_services/services_templates.md index ac6b85cc801..8b2c85802de 100644 --- a/doc/project_services/services_templates.md +++ b/doc/project_services/services_templates.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/services_templates.md' --- -This document was moved to [user/project/integrations/services_templates.md](../user/project/integrations/services_templates.md). +This document was moved to [another location](../user/project/integrations/services_templates.md). diff --git a/doc/project_services/slack.md b/doc/project_services/slack.md index 4c89ce92002..815032a08d5 100644 --- a/doc/project_services/slack.md +++ b/doc/project_services/slack.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/slack.md' --- -This document was moved to [user/project/integrations/slack.md](../user/project/integrations/slack.md). +This document was moved to [another location](../user/project/integrations/slack.md). diff --git a/doc/project_services/slack_slash_commands.md b/doc/project_services/slack_slash_commands.md index ca0034256f1..caae4d2ba4b 100644 --- a/doc/project_services/slack_slash_commands.md +++ b/doc/project_services/slack_slash_commands.md @@ -2,4 +2,4 @@ redirect_to: '../user/project/integrations/slack_slash_commands.md' --- -This document was moved to [user/project/integrations/slack_slash_commands.md](../user/project/integrations/slack_slash_commands.md). +This document was moved to [another location](../user/project/integrations/slack_slash_commands.md). diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index d04428fdbfe..dc6ee9b2503 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -6,13 +6,13 @@ type: reference GitLab allows [Owners](../user/permissions.md) to set a projects' visibility as **public**, **internal** or **private**. These visibility levels affect who can see the project in the -public access directory (`/public` under your GitLab instance), like at [https://gitlab.com/public](). +public access directory (`/public` under your GitLab instance), like at <https://gitlab.com/public> ## Visibility of projects ### Public projects -Public projects can be cloned **without any** authentication. +Public projects can be cloned **without any** authentication over https. They will be listed in the public access directory (`/public`) for all users. @@ -37,7 +37,7 @@ visibility setting keep this setting. You can read more about the change in the ### Private projects -Private projects can only be cloned and viewed by project members. +Private projects can only be cloned and viewed by project members (except for guests). They will appear in the public access directory (`/public`) for project members only. diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index dcc96507676..ad86555fc17 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -12,7 +12,7 @@ comments: false - [General Maintenance](../administration/raketasks/maintenance.md) and self-checks - [User management](user_management.md) - [Webhooks](web_hooks.md) -- [Import](import.md) of git repositories in bulk +- [Import](import.md) of Git repositories in bulk - [Rebuild authorized_keys file](../administration/raketasks/maintenance.md#rebuild-authorized_keys-file) task for administrators - [Migrate Uploads](../administration/raketasks/uploads/migrate.md) - [Sanitize Uploads](../administration/raketasks/uploads/sanitize.md) diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 092b4375208..336000f6cb9 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -77,7 +77,7 @@ You are highly advised to [read about storing configuration files](#storing-conf Use this command if you've installed GitLab with the Omnibus package: ```sh -sudo gitlab-rake gitlab:backup:create +sudo gitlab-backup create ``` Use this if you've installed GitLab from source: @@ -89,12 +89,12 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production If you are running GitLab within a Docker container, you can run the backup from the host: ```sh -docker exec -t <container name> gitlab-rake gitlab:backup:create +docker exec -t <container name> gitlab-backup create ``` If you are using the [GitLab helm chart](https://gitlab.com/charts/gitlab) on a Kubernetes cluster, you can run the backup task using `backup-utility` script on -the gitlab task runner pod via `kubectl`. Refer to [backing up a GitLab installation](https://gitlab.com/charts/gitlab/blob/master/doc/backup-restore/backup.md#backing-up-a-gitlab-installation) for more details: +the GitLab task runner pod via `kubectl`. Refer to [backing up a GitLab installation](https://gitlab.com/charts/gitlab/blob/master/doc/backup-restore/backup.md#backing-up-a-gitlab-installation) for more details: ```sh kubectl exec -it <gitlab task-runner pod> backup-utility @@ -199,7 +199,7 @@ To use the `copy` strategy instead of the default streaming strategy, specify `STRATEGY=copy` in the Rake task command. For example: ```sh -sudo gitlab-rake gitlab:backup:create STRATEGY=copy +sudo gitlab-backup create STRATEGY=copy ``` ### Backup filename @@ -207,7 +207,7 @@ sudo gitlab-rake gitlab:backup:create STRATEGY=copy By default a backup file is created according to the specification in [the Backup timestamp](#backup-timestamp) section above. You can however override the `[TIMESTAMP]` part of the filename by setting the `BACKUP` environment variable. For example: ```sh -sudo gitlab-rake gitlab:backup:create BACKUP=dump +sudo gitlab-backup create BACKUP=dump ``` The resulting file will then be `dump_gitlab_backup.tar`. This is useful for systems that make use of rsync and incremental backups, and will result in considerably faster transfer speeds. @@ -219,7 +219,7 @@ To make sure the generated archive is intelligently transferable by rsync, the ` Note that the `--rsyncable` option in `gzip` is not guaranteed to be available on all distributions. To verify that it is available in your distribution you can run `gzip --help` or consult the man pages. ```sh -sudo gitlab-rake gitlab:backup:create BACKUP=dump GZIP_RSYNCABLE=yes +sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes ``` ### Excluding specific directories from the backup @@ -244,7 +244,7 @@ will be skipped during a backup. For Omnibus GitLab packages: ```sh -sudo gitlab-rake gitlab:backup:create SKIP=db,uploads +sudo gitlab-backup create SKIP=db,uploads ``` For installations from source: @@ -315,8 +315,6 @@ if you see `411 Length Required` errors after attempting to upload, you may need to downgrade the `aws_signature_version` value from the default value to 2 [due to this issue](https://github.com/fog/fog-aws/issues/428). ---- - For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: @@ -428,8 +426,6 @@ For Omnibus GitLab packages: 1. [Reconfigure GitLab] for the changes to take effect ---- - For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: @@ -452,8 +448,8 @@ Note: This option only works for remote storage. If you want to group your backu you can pass a `DIRECTORY` environment variable: ``` -sudo gitlab-rake gitlab:backup:create DIRECTORY=daily -sudo gitlab-rake gitlab:backup:create DIRECTORY=weekly +sudo gitlab-backup create DIRECTORY=daily +sudo gitlab-backup create DIRECTORY=weekly ``` ### Uploading to locally mounted shares @@ -462,7 +458,7 @@ You may also send backups to a mounted share (`NFS` / `CIFS` / `SMB` / etc.) by using the Fog [`Local`](https://github.com/fog/fog-local#usage) storage provider. The directory pointed to by the `local_root` key **must** be owned by the `git` user **when mounted** (mounting with the `uid=` of the `git` user for `CIFS` and -`SMB`) or the user that you are executing the backup tasks under (for omnibus +`SMB`) or the user that you are executing the backup tasks under (for Omnibus packages, this is the `git` user). The `backup_upload_remote_directory` **must** be set in addition to the @@ -490,8 +486,6 @@ For Omnibus GitLab packages: 1. [Reconfigure GitLab] for the changes to take effect. ---- - For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: @@ -513,7 +507,7 @@ For installations from source: ### Backup archive permissions The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) -will have owner/group git:git and 0600 permissions by default. +will have owner/group `git`/`git` and 0600 permissions by default. This is meant to avoid other system users reading GitLab's data. If you need the backup archives to have different permissions you can use the 'archive_permissions' setting. @@ -527,8 +521,6 @@ For Omnibus GitLab packages: 1. [Reconfigure GitLab] for the changes to take effect. ---- - For installations from source: 1. Edit `/home/git/gitlab/config/gitlab.yml`: @@ -574,14 +566,12 @@ crontab -e There, add the following line to schedule the backup for everyday at 2 AM: ``` -0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1 +0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1 ``` You may also want to set a limited lifetime for backups to prevent regular backups using all your disk space. ---- - For installations from source: 1. Edit `home/git/gitlab/config/gitlab.yml`: @@ -624,7 +614,7 @@ GitLab that you created it on, for example CE 9.1.0. You need to have a working GitLab installation before you can perform a restore. This is mainly because the system user performing the -restore actions ('git') is usually not allowed to create or delete +restore actions (`git`) is usually not allowed to create or delete the SQL database it needs to import data into ('gitlabhq_production'). All existing data will be either erased (SQL) or moved to a separate directory (repositories, uploads). @@ -736,14 +726,15 @@ restore: ```shell # This command will overwrite the contents of your GitLab database! -sudo gitlab-rake gitlab:backup:restore BACKUP=1493107454_2018_04_25_10.6.4-ce +sudo gitlab-backup restore BACKUP=1493107454_2018_04_25_10.6.4-ce ``` Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary as mentioned above. -Restart and check GitLab: +Reconfigure, restart and check GitLab: ```shell +sudo gitlab-ctl reconfigure sudo gitlab-ctl restart sudo gitlab-rake gitlab:check SANITIZE=true ``` @@ -769,7 +760,7 @@ backup location (default location is `/var/opt/gitlab/backups`). For docker installations, the restore task can be run from host: ```sh -docker exec -it <name of container> gitlab-rake gitlab:backup:restore +docker exec -it <name of container> gitlab-backup restore ``` The GitLab helm chart uses a different process, documented in @@ -782,14 +773,14 @@ In this case you can consider using filesystem snapshots as part of your backup Example: Amazon EBS -> A GitLab server using omnibus-gitlab hosted on Amazon AWS. +> A GitLab server using Omnibus GitLab hosted on Amazon AWS. > An EBS drive containing an ext4 filesystem is mounted at `/var/opt/gitlab`. > In this case you could make an application backup by taking an EBS snapshot. > The backup includes all repositories, uploads and Postgres data. Example: LVM snapshots + rsync -> A GitLab server using omnibus-gitlab, with an LVM logical volume mounted at `/var/opt/gitlab`. +> A GitLab server using Omnibus GitLab, with an LVM logical volume mounted at `/var/opt/gitlab`. > Replicating the `/var/opt/gitlab` directory using rsync would not be reliable because too many files would change while rsync is running. > Instead of rsync-ing `/var/opt/gitlab`, we create a temporary LVM snapshot, which we mount as a read-only filesystem at `/mnt/gitlab_backup`. > Now we can have a longer running rsync job which will create a consistent replica on the remote server. @@ -813,7 +804,8 @@ will have all your repositories, but not any other data. ## Troubleshooting -### Restoring database backup using omnibus packages outputs warnings +### Restoring database backup using Omnibus packages outputs warnings + If you are using backup restore procedures you might encounter the following warnings: ``` @@ -842,7 +834,7 @@ including (but not restricted to): - [CI/CD variables](../ci/variables/README.md) - [Kubernetes / GCP integration](../user/project/clusters/index.md) -- [Custom Pages domains](../user/project/pages/getting_started_part_three.md) +- [Custom Pages domains](../user/project/pages/custom_domains_ssl_tls_certification/index.md) - [Project error tracking](../user/project/operations/error_tracking.md) - [Runner authentication](../ci/runners/README.md) - [Project mirroring](../workflow/repository_mirroring.md) @@ -931,6 +923,29 @@ backup beforehand. UPDATE ci_runners SET token = null, token_encrypted = null; ``` +#### Reset pending pipeline jobs + +1. Enter the DB console: + + For Omnibus GitLab packages: + + ```sh + sudo gitlab-rails dbconsole + ``` + + For installations from source: + + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` + +1. Clear all the tokens for pending jobs: + + ```sql + -- Clear build tokens + UPDATE ci_builds SET token = null, token_encrypted = null; + ``` + A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted, GitLab can be brought back into working order, and the lost data can be manually replaced. @@ -968,3 +983,21 @@ want to run the chown against your custom location instead of [reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure [restart GitLab]: ../administration/restart_gitlab.md#installations-from-source + +### Backup fails to complete with Gzip error + +While running the backup, you may receive a gzip error: + +```sh +sudo /opt/gitlab/bin/gitlab-backup create +Dumping ... +... +gzip: stdout: Input/output error + +Backup failed +``` + +If this happens, check the following: + +1. Confirm there is sufficent diskspace for the gzip operation. +1. If NFS is being used, check if the mount option `timeo` is set. The default is `600`, and changing this to smaller values have resulted in this error. diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index f880f31c39e..f84d29cca9a 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -1,6 +1,10 @@ # Cleanup -## Remove garbage from filesystem. Important! Data loss! +## Remove garbage from filesystem + +DANGER: **Danger:** +The commands below will remove data permanently from your GitLab instance. Only use +these commands if you are 100% certain that it is safe to delete this data. Remove namespaces(dirs) from all repository storage paths if they don't exist in GitLab database. @@ -137,3 +141,13 @@ level with `NICENESS`. Below are the valid levels, but consult - `1` or `Realtime` - `2` or `Best-effort` (default) - `3` or `Idle` + +## Remove expired ActiveSession lookup keys + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:cleanup:sessions:active_sessions_lookup_keys + +# installation from source +bundle exec rake gitlab:cleanup:sessions:active_sessions_lookup_keys RAILS_ENV=production +``` diff --git a/doc/raketasks/features.md b/doc/raketasks/features.md index 57c16f110e4..c06800a2aa3 100644 --- a/doc/raketasks/features.md +++ b/doc/raketasks/features.md @@ -6,7 +6,7 @@ This command will enable the namespaces feature introduced in v4.0. It will move Note: -- Because the **repository location will change**, you will need to **update all your git URLs** to point to the new location. +- Because the **repository location will change**, you will need to **update all your Git URLs** to point to the new location. - Username can be changed at **Profile ➔ Account**. **Example:** diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index b59c06a24ea..d93e7241fda 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -11,17 +11,17 @@ ## How to use -### Create a new folder to import your Git repositories from. +### Create a new folder to import your Git repositories from -The new folder needs to have git user ownership and read/write/execute access for git user and its group: +The new folder needs to have Git user ownership and read/write/execute access for Git user and its group: ``` sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group ``` -### Copy your bare repositories inside this newly created folder: +### Copy your bare repositories inside this newly created folder -- Any .git repositories found on any of the subfolders will be imported as projects +- Any `.git` repositories found on any of the subfolders will be imported as projects - Groups will be created as needed, these could be nested folders. Example: If we copy the repos to `/var/opt/gitlab/git-data/repository-import-<date>`, and repo A needs to be under the groups G1 and G2, it will @@ -34,11 +34,11 @@ sudo cp -r /old/git/foo.git /var/opt/gitlab/git-data/repository-import-<date>/ne sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-<date> ``` -`foo.git` needs to be owned by the git user and git users group. +`foo.git` needs to be owned by the `git` user and `git` users group. If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`. -### Run the command below depending on your type of installation: +### Run the command below depending on your type of installation #### Omnibus Installation @@ -96,11 +96,11 @@ Importing bare repositories from hashed storage is unsupported. To support importing bare repositories from hashed storage, GitLab 10.4 and later stores the full project path with each repository, in a special section of -the git repository's config file. This section is formatted as follows: +the Git repository's config file. This section is formatted as follows: ``` [gitlab] - fullpath = gitlab-org/gitlab-ce + fullpath = gitlab-org/gitlab-ce ``` However, existing repositories were not migrated to include this path. diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index a498e9793c1..d9a042ec8fe 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -1,6 +1,6 @@ # Webhooks administration **(CORE ONLY)** -## Add a webhook for **ALL** projects: +## Add a webhook for **ALL** projects ```sh # omnibus-gitlab @@ -9,7 +9,7 @@ sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" RAILS_ENV=production ``` -## Add a webhook for projects in a given **NAMESPACE**: +## Add a webhook for projects in a given **NAMESPACE** ```sh # omnibus-gitlab @@ -18,7 +18,7 @@ sudo gitlab-rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acm bundle exec rake gitlab:web_hook:add URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ``` -## Remove a webhook from **ALL** projects using: +## Remove a webhook from **ALL** projects using ```sh # omnibus-gitlab @@ -27,7 +27,7 @@ sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" RAILS_ENV=production ``` -## Remove a webhook from projects in a given **NAMESPACE**: +## Remove a webhook from projects in a given **NAMESPACE** ```sh # omnibus-gitlab @@ -36,7 +36,7 @@ sudo gitlab-rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme bundle exec rake gitlab:web_hook:rm URL="http://example.com/hook" NAMESPACE=acme RAILS_ENV=production ``` -## List **ALL** webhooks: +## List **ALL** webhooks ```sh # omnibus-gitlab @@ -45,7 +45,7 @@ sudo gitlab-rake gitlab:web_hook:list bundle exec rake gitlab:web_hook:list RAILS_ENV=production ``` -## List the webhooks from projects in a given **NAMESPACE**: +## List the webhooks from projects in a given **NAMESPACE** ```sh # omnibus-gitlab @@ -53,3 +53,8 @@ sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme # source installations bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production ``` + +## Local requests in webhooks + +[Requests to local network by webhooks](../security/webhooks.md) can be allowed +or blocked by an administrator. diff --git a/doc/security/README.md b/doc/security/README.md index c48d5bc2065..e3fb07c69c2 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -5,9 +5,10 @@ type: index # Security +- [Password storage](password_storage.md) - [Password length limits](password_length_limits.md) - [Restrict SSH key technologies and minimum length](ssh_keys_restrictions.md) -- [Rack attack](rack_attack.md) +- [Rate limits](rate_limits.md) - [Webhooks and insecure internal web services](webhooks.md) - [Information exclusivity](information_exclusivity.md) - [Reset your root password](reset_root_password.md) diff --git a/doc/security/img/outbound_requests_section.png b/doc/security/img/outbound_requests_section.png Binary files differdeleted file mode 100644 index f7783f34cdd..00000000000 --- a/doc/security/img/outbound_requests_section.png +++ /dev/null diff --git a/doc/security/img/outbound_requests_section_v12_2.png b/doc/security/img/outbound_requests_section_v12_2.png Binary files differnew file mode 100644 index 00000000000..3dc99868a35 --- /dev/null +++ b/doc/security/img/outbound_requests_section_v12_2.png diff --git a/doc/security/img/whitelist.png b/doc/security/img/whitelist.png Binary files differnew file mode 100644 index 00000000000..897000e804d --- /dev/null +++ b/doc/security/img/whitelist.png diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index 62a20d3f257..749ccf924b5 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -1,6 +1,7 @@ --- type: concepts --- + # Information exclusivity Git is a distributed version control system (DVCS). This means that everyone diff --git a/doc/security/password_length_limits.md b/doc/security/password_length_limits.md index d78293c75c6..9909ef4a8e4 100644 --- a/doc/security/password_length_limits.md +++ b/doc/security/password_length_limits.md @@ -1,19 +1,31 @@ --- type: reference, howto --- + # Custom password length limits -If you want to enforce longer user passwords you can create an extra Devise -initializer with the steps below. +The user password length is set to a minimum of 8 characters by default. +To change that for installations from source: + +1. Edit `devise_password_length.rb`: + + ```sh + cd /home/git/gitlab + sudo -u git -H cp config/initializers/devise_password_length.rb.example config/initializers/devise_password_length.rb + sudo -u git -H editor config/initializers/devise_password_length.rb + ``` + +1. Change the new password length limits: + + ```ruby + config.password_length = 12..128 + ``` -If you do not use the `devise_password_length.rb` initializer the password -length is set to a minimum of 8 characters in `config/initializers/devise.rb`. + In this example, the minimum length is 12 characters, and the maximum length + is 128 characters. -```bash -cd /home/git/gitlab -sudo -u git -H cp config/initializers/devise_password_length.rb.example config/initializers/devise_password_length.rb -sudo -u git -H editor config/initializers/devise_password_length.rb # inspect and edit the new password length limits -``` +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. <!-- ## Troubleshooting diff --git a/doc/security/password_storage.md b/doc/security/password_storage.md new file mode 100644 index 00000000000..f4e32f96f7b --- /dev/null +++ b/doc/security/password_storage.md @@ -0,0 +1,13 @@ +--- +type: reference +--- + +# Password Storage + +GitLab stores user passwords in a hashed format, to prevent passwords from being visible. + +GitLab uses the [Devise](https://github.com/plataformatec/devise) authentication library, which handles the hashing of user passwords. Password hashes are created with the following attributes: + +- **Hashing**: the [bcrypt](https://en.wikipedia.org/wiki/Bcrypt) hashing function is used to generate the hash of the provided password. This is a strong, industry-standard cryptographic hashing function. +- **Stretching**: Password hashes are [stretched](https://en.wikipedia.org/wiki/Key_stretching) to harden against brute-force attacks. GitLab uses a stretching factor of 10 by default. +- **Salting**: A [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)) is added to each password to harden against pre-computed hash and dictionary attacks. Each salt is randomly generated for each password, so that no two passwords share a salt, to further increase security. diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index 8695b5d2194..b99bfb16829 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -1,7 +1,10 @@ --- type: reference, howto --- -# Rack Attack + +# Rack Attack initializer + +## Overview [Rack Attack](https://github.com/kickstarter/rack-attack), also known as Rack::Attack, is a Ruby gem that is meant to protect GitLab with the ability to customize throttling and @@ -13,19 +16,72 @@ If you find throttling is not enough to protect you against abusive clients, Rack Attack offers IP whitelisting, blacklisting, Fail2ban style filtering, and tracking. -**Note:** Starting with 11.2, Rack Attack is disabled by default. To continue -using Rack Attack, please enable it by [configuring `gitlab.rb` as described in Settings](#settings). +For more information on how to use these options see the [Rack Attack README](https://github.com/kickstarter/rack-attack/blob/master/README.md). + +NOTE: **Note:** See +[User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md) +for simpler limits that are configured in the UI. + +NOTE: **Note:** Starting with GitLab 11.2, Rack Attack is disabled by default. If your +instance is not exposed to the public internet, it is recommended that you leave +Rack Attack disabled. + +## Behavior + +If set up as described in the [Settings](#settings) section below, two behaviors +will be enabled: + +- Protected paths will be throttled. +- Failed authentications for Git and container registry requests will trigger a temporary IP ban. + +### Protected paths throttle + +GitLab responds with HTTP status code `429` to POST requests at protected paths +that exceed 10 requests per minute per IP address. + +By default, protected paths are: + +```ruby +default['gitlab']['gitlab-rails']['rack_attack_protected_paths'] = [ + '/users/password', + '/users/sign_in', + '/api/#{API::API.version}/session.json', + '/api/#{API::API.version}/session', + '/users', + '/users/confirmation', + '/unsubscribes/', + '/import/github/personal_access_token' +] +``` + +This header is included in responses to blocked requests: + +``` +Retry-After: 60 +``` + +For example, the following are limited to a maximum 10 requests per minute: + +- User sign-in +- User sign-up (if enabled) +- User password reset + +After 10 requests, the client must wait a minute before it can +try again. + +### Git and container registry failed authentication ban + +GitLab responds with HTTP status code `403` for 1 hour, if 30 failed +authentication requests were received in a 3-minute period from a single IP address. -By default, user sign-in, user sign-up (if enabled), and user password reset is -limited to 6 requests per minute. After trying for 6 times, the client will -have to wait for the next minute to be able to try again. +This applies only to Git requests and container registry (`/jwt/auth`) requests +(combined). -If you installed or upgraded GitLab by following the [official guides](../install/README.md), -Rack Attack should be disabled by default. If your instance is not exposed to any incoming -connections, it is recommended that you leave Rack Attack disabled. +This limit is reset by requests that authenticate successfully. For example, 29 +failed authentication requests followed by 1 successful request, followed by 29 +more failed authentication requests would not trigger a ban. -For more information on how to use these options check out -[rack-attack README](https://github.com/kickstarter/rack-attack/blob/master/README.md). +No response headers are provided. ## Settings @@ -34,34 +90,34 @@ For more information on how to use these options check out 1. Open `/etc/gitlab/gitlab.rb` with your editor 1. Add the following: - ```ruby - gitlab_rails['rack_attack_git_basic_auth'] = { - 'enabled' => true, - 'ip_whitelist' => ["127.0.0.1"], - 'maxretry' => 10, # Limit the number of Git HTTP authentication attempts per IP - 'findtime' => 60, # Reset the auth attempt counter per IP after 60 seconds - 'bantime' => 3600 # Ban an IP for one hour (3600s) after too many auth attempts - } - ``` + ```ruby + gitlab_rails['rack_attack_git_basic_auth'] = { + 'enabled' => true, + 'ip_whitelist' => ["127.0.0.1"], + 'maxretry' => 10, # Limit the number of Git HTTP authentication attempts per IP + 'findtime' => 60, # Reset the auth attempt counter per IP after 60 seconds + 'bantime' => 3600 # Ban an IP for one hour (3600s) after too many auth attempts + } + ``` 1. Reconfigure GitLab: - ``` - sudo gitlab-ctl reconfigure - ``` + ``` + sudo gitlab-ctl reconfigure + ``` The following settings can be configured: - `enabled`: By default this is set to `false`. Set this to `true` to enable Rack Attack. - `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a Ruby array. - CIDR notation is supported in GitLab v12.1 and up. - For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`. + CIDR notation is supported in GitLab v12.1 and up. + For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`. - `maxretry`: The maximum amount of times a request can be made in the - specified time. + specified time. - `findtime`: The maximum amount of time that failed requests can count against an IP - before it's blacklisted (in seconds). + before it's blacklisted (in seconds). - `bantime`: The total amount of time that a blacklisted IP will be blocked (in - seconds). + seconds). **Installations from source** @@ -71,25 +127,25 @@ taken in order to enable protection for your GitLab instance: 1. In `config/application.rb` find and uncomment the following line: - ```ruby - config.middleware.use Rack::Attack - ``` + ```ruby + config.middleware.use Rack::Attack + ``` 1. Copy `config/initializers/rack_attack.rb.example` to `config/initializers/rack_attack.rb` 1. Open `config/initializers/rack_attack.rb`, review the `paths_to_be_protected`, and add any other path you need protecting 1. Restart GitLab: - ```sh - sudo service gitlab restart - ``` + ```sh + sudo service gitlab restart + ``` If you want more restrictive/relaxed throttle rules, edit `config/initializers/rack_attack.rb` and change the `limit` or `period` values. For example, more relaxed throttle rules will be if you set `limit: 3` and `period: 1.seconds` (this will allow 3 requests per second). You can also add other paths to the protected list by adding to `paths_to_be_protected` -variable. If you change any of these settings do not forget to restart your +variable. If you change any of these settings you must restart your GitLab instance. ## Remove blocked IPs from Rack Attack via Redis @@ -98,28 +154,28 @@ In case you want to remove a blocked IP, follow these steps: 1. Find the IPs that have been blocked in the production log: - ```sh - grep "Rack_Attack" /var/log/gitlab/gitlab-rails/auth.log - ``` + ```sh + grep "Rack_Attack" /var/log/gitlab/gitlab-rails/auth.log + ``` 1. Since the blacklist is stored in Redis, you need to open up `redis-cli`: - ```sh - /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket - ``` + ```sh + /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket + ``` 1. You can remove the block using the following syntax, replacing `<ip>` with the actual IP that is blacklisted: - ``` - del cache:gitlab:rack::attack:allow2ban:ban:<ip> - ``` + ``` + del cache:gitlab:rack::attack:allow2ban:ban:<ip> + ``` 1. Confirm that the key with the IP no longer shows up: - ``` - keys *rack::attack* - ``` + ``` + keys *rack::attack* + ``` 1. Optionally, add the IP to the whitelist to prevent it from being blacklisted again (see [settings](#settings)). @@ -136,8 +192,8 @@ the load balancer. In that case, you will need to: 1. Whitelist the load balancer's IP address(es) in the Rack Attack [settings](#settings). 1. Reconfigure GitLab: - ``` - sudo gitlab-ctl reconfigure - ``` + ``` + sudo gitlab-ctl reconfigure + ``` 1. [Remove the block via Redis.](#remove-blocked-ips-from-rack-attack-via-redis) diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md new file mode 100644 index 00000000000..80088da77a0 --- /dev/null +++ b/doc/security/rate_limits.md @@ -0,0 +1,32 @@ +--- +type: reference, howto +--- + +# Rate limits + +NOTE: **Note:** +For GitLab.com, please see +[GitLab.com-specific rate limits](../user/gitlab_com/index.md#gitlabcom-specific-rate-limits). + +Rate limiting is a common technique used to improve the security and durability +of a web application. + +For example, a simple script can make thousands of web requests per second. +Whether malicious, apathetic, or just a bug, your application and infrastructure +may not be able to cope with the load. For more details, see +[Denial-of-service attack](https://en.wikipedia.org/wiki/Denial-of-service_attack). +Most cases can be mitigated by limiting the rate of requests from a single IP address. + +Most [brute-force attacks](https://en.wikipedia.org/wiki/Brute-force_attack) are +similarly mitigated by a rate limit. + +## Admin Area settings + +- [User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md). +- [Rate limits on raw endpoints](../user/admin_area/settings/rate_limits_on_raw_endpoints.md) + +## Rack Attack initializer + +This method of rate limiting is cumbersome, but has some advantages. It allows +throttling of specific paths, and is also integrated into Git and container +registry requests. See [Rack Attack initializer](rack_attack.md). diff --git a/doc/security/reset_root_password.md b/doc/security/reset_root_password.md index a58d70f0ff2..ec360e2d338 100644 --- a/doc/security/reset_root_password.md +++ b/doc/security/reset_root_password.md @@ -1,6 +1,7 @@ --- type: howto --- + # How to reset your root password To reset your root password, first log into your server with root privileges. @@ -22,7 +23,7 @@ user = User.where(id: 1).first or ```bash -user = User.find_by(email: 'admin@local.host') +user = User.find_by(email: 'admin@example.com') ``` Now you can change your password: diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index ae4cc44519e..4c60daf77f4 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -1,6 +1,7 @@ --- type: reference, howto --- + # Restrict allowed SSH key technologies and minimum length `ssh-keygen` allows users to create RSA keys with as few as 768 bits, which diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index 49dadd5abc2..b08d9ffa26e 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -1,6 +1,7 @@ --- type: howto --- + # Enforce Two-factor Authentication (2FA) Two-factor Authentication (2FA) provides an additional level of security to your @@ -48,11 +49,11 @@ need to be administrator or owner of the group. The following are important notes about 2FA: - Projects belonging to a 2FA-enabled group that - [is shared](../user/project/members/share_project_with_groups.md) + [is shared](../user/project/members/share_project_with_groups.md) with a 2FA-disabled group will *not* require members of the 2FA-disabled group to use - 2FA for the project. For example, if project *P* belongs to 2FA-enabled group *A* and + 2FA for the project. For example, if project *P* belongs to 2FA-enabled group *A* and is shared with 2FA-disabled group *B*, members of group *B* can access project *P* - without 2FA. To ensure this scenario doesn't occur, + without 2FA. To ensure this scenario doesn't occur, [prevent sharing of projects](../user/group/index.md#share-with-group-lock) for the 2FA-enabled group. - If you add additional members to a project within a group or subgroup that has diff --git a/doc/security/unlock_user.md b/doc/security/unlock_user.md index 75cf754e197..d34826c853c 100644 --- a/doc/security/unlock_user.md +++ b/doc/security/unlock_user.md @@ -1,38 +1,45 @@ --- type: howto --- -# How to unlock a locked user -To unlock a locked user, first log into your server with root privileges. +# How to unlock a locked user from the command line -Start a Ruby on Rails console with this command: +After six failed login attempts a user gets in a locked state. +To unlock a locked user: -```bash -gitlab-rails console production -``` +1. SSH into your GitLab server. +1. Start a Ruby on Rails console: -Wait until the console has loaded. + ```sh + ## For Omnibus GitLab + sudo gitlab-rails console production -There are multiple ways to find your user. You can search for email or username. + ## For installations from source + sudo -u git -H bundle exec rails console RAILS_ENV=production + ``` -```bash -user = User.where(id: 1).first -``` +1. Find the user to unlock. You can search by email or ID. -or + ```ruby + user = User.find_by(email: 'admin@local.host') + ``` -```bash -user = User.find_by(email: 'admin@local.host') -``` + or -Unlock the user: + ```ruby + user = User.where(id: 1).first + ``` -```bash -user.unlock_access! -``` +1. Unlock the user: -Exit the console, the user should now be able to log in again. + ```ruby + user.unlock_access! + ``` + +1. Exit the console with <kbd>Ctrl</kbd>+<kbd>d</kbd> + +The user should now be able to log in. <!-- ## Troubleshooting diff --git a/doc/security/user_email_confirmation.md b/doc/security/user_email_confirmation.md index f0af0a7ac6a..7ba50acbb06 100644 --- a/doc/security/user_email_confirmation.md +++ b/doc/security/user_email_confirmation.md @@ -1,6 +1,7 @@ --- type: howto --- + # User email confirmation at sign-up GitLab can be configured to require confirmation of a user's email address when diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md index 00a2607b607..9fc8f7ec985 100644 --- a/doc/security/user_file_uploads.md +++ b/doc/security/user_file_uploads.md @@ -1,6 +1,7 @@ --- type: reference --- + # User File Uploads Images that are attached to issues, merge requests, or comments @@ -12,7 +13,7 @@ image contains sensitive information. Authentication is not enabled because images must be visible in the body of notification emails, which are often read from email clients that are not authenticated with GitLab, such as Outlook, Apple Mail, or the Mail app on your -mobile device. +mobile device. >**Note:** Non-image attachments do require authentication to be viewed. diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index d4fa088cb15..e39bc9a9626 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -1,6 +1,7 @@ --- type: concepts, reference, howto --- + # Webhooks and insecure internal web services If you have non-GitLab web services running on your GitLab server or within its @@ -33,15 +34,46 @@ to 127.0.0.1, ::1 and 0.0.0.0, as well as IPv4 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and IPv6 site-local (ffc0::/10) addresses won't be allowed. This behavior can be overridden by enabling the option *"Allow requests to the -local network from hooks and services"* in the *"Outbound requests"* section +local network from web hooks and services"* in the *"Outbound requests"* section inside the Admin area under **Settings** (`/admin/application_settings/network`): - + + +NOTE: **Note:** +*System hooks* are enabled to make requests to local network by default since they are +set up by administrators. However, you can turn this off by disabling the +**Allow requests to the local network from system hooks** option. + +## Whitelist for local requests + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/44496) in GitLab 12.2 + +You can allow certain domains and IP addresses to be accessible to both *system hooks* +and *webhooks* even when local requests are not allowed by adding them to the +whitelist. Navigate to **Admin Area > Settings > Network** (`/admin/application_settings/network`) +and expand **Outbound requests**: + + + +The whilelist entries can be separated by semicolons, commas or whitespaces +(including newlines) and be in different formats like hostnames, IP addresses and/or +IP ranges. IPv6 is supported. Hostnames that contain unicode characters should +use IDNA encoding. + +The whitelist can hold a maximum of 1000 entries. Each entry can be a maximum of +255 characters. + +Example: + +```text +example.com;gitlab.example.com +127.0.0.1,1:0:0:0:0:0:0:1 +127.0.0.0/8 1:0:0:0:0:0:0:0/124 +``` ->**Note:** -*System hooks* are exempt from this protection because they are set up by -admins. +NOTE: **Note:** +Wildcards (`*.example.com`) and ports (`127.0.0.1:3000`) are not currently supported. <!-- ## Troubleshooting diff --git a/doc/ssh/README.md b/doc/ssh/README.md index dbd9bcee935..e18f49de3f0 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -64,13 +64,13 @@ Following [best practices](https://linux-audit.com/using-ed25519-openssh-keys-in you should always favor [ED25519](https://ed25519.cr.yp.to/) SSH keys, since they are more secure and have better performance over the other types. -ED25519 SSH keys were introduced in OpenSSH 6.5, -so any modern OS should include the option to create them. -If for any reason your OS or the GitLab instance you interact with doesn't +ED25519 SSH keys were introduced in OpenSSH 6.5, +so any modern OS should include the option to create them. +If for any reason your OS or the GitLab instance you interact with doesn't support ED25519, you can fallback to RSA. NOTE: **Note:** -Omnibus does not ship with OpenSSH, so it uses the version on your GitLab server. If using +Omnibus does not ship with OpenSSH, so it uses the version on your GitLab server. If using Omnibus, ensure the version of OpenSSH installed is version 6.5 or newer if you want to use ED25519 SSH keys. ### RSA SSH keys @@ -107,18 +107,18 @@ To create a new SSH key pair: 1. Open a terminal on Linux or macOS, or Git Bash / WSL on Windows. 1. Generate a new ED25519 SSH key pair: - ```bash - ssh-keygen -t ed25519 -C "email@example.com" - ``` + ```bash + ssh-keygen -t ed25519 -C "email@example.com" + ``` - Or, if you want to use RSA: + Or, if you want to use RSA: - ```bash - ssh-keygen -o -t rsa -b 4096 -C "email@example.com" - ``` + ```bash + ssh-keygen -o -t rsa -b 4096 -C "email@example.com" + ``` - The `-C` flag adds a comment in the key in case you have multiple of them - and want to tell which is which. It is optional. + The `-C` flag adds a comment in the key in case you have multiple of them + and want to tell which is which. It is optional. 1. Next, you will be prompted to input a file path to save your SSH key pair to. If you don't already have an SSH key pair and aren't generating a [deploy key](#deploy-keys), @@ -126,21 +126,21 @@ To create a new SSH key pair: <kbd>Enter</kbd>. Using the suggested path will normally allow your SSH client to automatically use the SSH key pair with no additional configuration. - If you already have an SSH key pair with the suggested file path, you will need - to input a new file path and [declare what host](#working-with-non-default-ssh-key-pair-paths) - this SSH key pair will be used for in your `~/.ssh/config` file. + If you already have an SSH key pair with the suggested file path, you will need + to input a new file path and [declare what host](#working-with-non-default-ssh-key-pair-paths) + this SSH key pair will be used for in your `~/.ssh/config` file. 1. Once the path is decided, you will be prompted to input a password to secure your new SSH key pair. It's a best practice to use a password, but it's not required and you can skip creating it by pressing <kbd>Enter</kbd> twice. - If, in any case, you want to add or change the password of your SSH key pair, - you can use the `-p` flag: + If, in any case, you want to add or change the password of your SSH key pair, + you can use the `-p` flag: - ``` - ssh-keygen -p -o -f <keyname> - ``` + ``` + ssh-keygen -p -o -f <keyname> + ``` Now, it's time to add the newly created public key to your GitLab account. @@ -149,41 +149,40 @@ Now, it's time to add the newly created public key to your GitLab account. 1. Copy your **public** SSH key to the clipboard by using one of the commands below depending on your Operating System: - **macOS:** + **macOS:** - ```bash - pbcopy < ~/.ssh/id_ed25519.pub - ``` + ```bash + pbcopy < ~/.ssh/id_ed25519.pub + ``` - **WSL / GNU/Linux (requires the xclip package):** + **WSL / GNU/Linux (requires the xclip package):** - ```bash - xclip -sel clip < ~/.ssh/id_ed25519.pub - ``` + ```bash + xclip -sel clip < ~/.ssh/id_ed25519.pub + ``` - **Git Bash on Windows:** + **Git Bash on Windows:** - ```bash - cat ~/.ssh/id_ed25519.pub | clip - ``` + ```bash + cat ~/.ssh/id_ed25519.pub | clip + ``` - You can also open the key in a graphical editor and copy it from there, - but be careful not to accidentally change anything. + You can also open the key in a graphical editor and copy it from there, + but be careful not to accidentally change anything. - NOTE: **Note:** - If you opted to create an RSA key, the name might differ. + NOTE: **Note:** + If you opted to create an RSA key, the name might differ. 1. Add your **public** SSH key to your GitLab account by: 1. Clicking your avatar in the upper right corner and selecting **Settings**. 1. Navigating to **SSH Keys** and pasting your **public** key in the **Key** field. If you: - - Created the key with a comment, this will appear in the **Title** field. - Created the key without a comment, give your key an identifiable title like _Work Laptop_ or _Home Workstation_. 1. Click the **Add key** button. - NOTE: **Note:** - If you manually copied your public SSH key make sure you copied the entire - key starting with `ssh-ed25519` (or `ssh-rsa`) and ending with your email. + NOTE: **Note:** + If you manually copied your public SSH key make sure you copied the entire + key starting with `ssh-ed25519` (or `ssh-rsa`) and ending with your email. ## Testing that everything is set up correctly diff --git a/doc/subscriptions/billing_table.png b/doc/subscriptions/billing_table.png Binary files differdeleted file mode 100644 index acd1b6193ec..00000000000 --- a/doc/subscriptions/billing_table.png +++ /dev/null diff --git a/doc/subscriptions/img/additional_minutes.png b/doc/subscriptions/img/additional_minutes.png Binary files differnew file mode 100644 index 00000000000..b159b98c9ce --- /dev/null +++ b/doc/subscriptions/img/additional_minutes.png diff --git a/doc/subscriptions/img/buy_btn.png b/doc/subscriptions/img/buy_btn.png Binary files differnew file mode 100644 index 00000000000..4fd05c0fba7 --- /dev/null +++ b/doc/subscriptions/img/buy_btn.png diff --git a/doc/subscriptions/img/buy_minutes_card.png b/doc/subscriptions/img/buy_minutes_card.png Binary files differnew file mode 100644 index 00000000000..cab098300cd --- /dev/null +++ b/doc/subscriptions/img/buy_minutes_card.png diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 745a2253e84..13c406727ab 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -2,118 +2,314 @@ type: index, reference --- -# Subscription setup and management +# Customer Docs -This page will help get you started with your new subscription or manage an existing one, whether you have subscribed to GitLab.com or self-managed GitLab. +This section contains information for: -To subscribe, upgrade, or read more about the types of subscriptions, please see [Subscribe to GitLab](../README.md#subscribe-to-gitlab) on the GitLab Documentation landing page. +- New customers about choosing [which GitLab](#which-gitlab) is right for you. +- Existing customers about [managing subscriptions](#managing-subscriptions). -## Set up GitLab +Also see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/). -Learn how GitLab helps you in the stages of the DevOps lifecycle by learning more [about the GitLab product](https://about.gitlab.com/product/), [GitLab features](https://about.gitlab.com/features/), and [GitLab Documentation](../README.md). +## Which GitLab? -### Self-managed: Install GitLab +There are two ways to use GitLab: -Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. +- [GitLab.com](#gitlabcom): GitLab's SaaS offering. You don't need to install + anything to use GitLab.com, you only need to + [sign up](https://gitlab.com/users/sign_in) and start using GitLab straight away. +- [GitLab self-managed](#gitlab-self-managed): Install, administer, and maintain + your own GitLab instance. -### GitLab.com: Create a user and group +The following sections outline tiers and features within GitLab.com +and GitLab self-managed. -Start with creating a user account for yourself using our [sign up page](https://gitlab.com/users/sign_in#register-pane). +### GitLab.com + +GitLab.com is hosted, managed, and administered by GitLab, Inc., with +[free and paid subscriptions](https://about.gitlab.com/pricing/) for individuals +and teams in the following tiers: -[GitLab groups](../user/group/index.md) help assemble related projects together allowing you to grant members access to several projects at once. A group is not required if you plan on having [projects](../user/project/) inside a personal namespace. +| Tier | Includes same features available in | +|:-------|:----------------------------------------------------| +| Free | [Core](#gitlab-self-managed) self-managed tier. | +| Bronze | [Starter](#gitlab-self-managed) self-managed tier. | +| Silver | [Premium](#gitlab-self-managed) self-managed tier. | +| Gold | [Ultimate](#gitlab-self-managed) self-managed tier. | -## Your subscription +GitLab.com subscriptions grant access +to the same features available in GitLab self-managed, **except +[administration](../administration/index.md) tools and settings**. -You can view and manage subscriptions through our [Customers portal](https://customers.gitlab.com/). Information on applying your subscription is below. +GitLab.com allows you to apply your subscription to a group or your personal user. -Please also see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/) +When applied to: -### View subscription and seats +- A **group**, the group, all subgroups, and all projects under the selected + group on GitLab.com will have the features of the associated plan. It is + recommended to go with a group plan when managing projects and users of an + organization. +- A **personal userspace** instead, all projects will have features with the + subscription applied, but as it is not a group, group features will not be available. -To view and manage the subscriptions you have purchased and the number of seats associated with the subscription, please visit and log into the [Customers’ Portal](https://customers.gitlab.com/subscriptions). For more information, please see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/) and [pricing page](https://about.gitlab.com/pricing/), which includes information on our [true-up pricing policy](https://about.gitlab.com/handbook/product/pricing/#true-up-pricing) when adding more users than at the time of purchase. +TIP: **Tip:** +To support the open source community and encourage the development of open +source projects, GitLab grants access to **Gold** features for all GitLab.com +**public** projects, regardless of the subscription. -Please note that this account may have the same email, but is a _separate_ login from your GitLab.com account. If the two accounts are linked together, then you can use the "sign in with GitLab.com account" link underneath the `Sign In` button. +The following resources are available for more information on GitLab.com: -### Change billing information +- [Feature comparison](https://about.gitlab.com/pricing/gitlab-com/feature-comparison/), for information on what features are available at each tier. +- [GitLab pricing page](https://about.gitlab.com/pricing/), for subscription information and a free trial. +- Our [product marketing page](https://about.gitlab.com/handbook/marketing/product-marketing/), for additional information including: + - How [different tiers are licensed](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers). + - The different [GitLab distributions](https://about.gitlab.com/handbook/marketing/product-marketing/#distributions). + +#### Subscribing to GitLab.com + +To subscribe to GitLab.com: + +1. Create a user account for yourself using our + [sign up page](https://gitlab.com/users/sign_in#register-pane). +1. Create a [group](../user/group/index.md). GitLab groups help assemble related + projects together allowing you to grant members access to several projects + at once. A group is not required if you plan on having projects inside a personal + namespace. +1. Create additional users and + [add them to the group](../user/group/index.md#add-users-to-a-group). +1. Select the **Bronze**, **Silver**, or **Gold** GitLab.com plan through the + [GitLab Subscription Manager](https://customers.gitlab.com/). +1. Link your GitLab.com account with your GitLab Subscription Manager account. + Once signed into the GitLab Subscription Manager, if your account is not + already linked, you will prompted to link your account with a + **Link my GitLab Account** button. +1. Associate the group with the subscription. + +TIP: **Tip:** +You can also go to the [**My Account**](https://customers.gitlab.com/customers/edit) +page to add or change the GitLab.com account link. + +### GitLab self-managed + +With GitLab self-managed, you deploy your own GitLab instance on-premises or on a cloud of your choice. +GitLab self-managed is available for [free and with paid subscriptions](https://about.gitlab.com/pricing/#self-managed) in the following tiers: + +| Tier | Includes | +|:---------|:-----------------------------------------------| +| Core | Core features. | +| Starter | Core and Starter features. | +| Premium | Core, Starter, and Premium features. | +| Ultimate | Core, Starter, Premium, and Ultimate features. | + +The following resources are available for more information on GitLab self-managed: -In the customers portal, go to the `My Account` page, then revise the `Account Details` information and click on the `Update Account` button. +- [Feature comparison](https://about.gitlab.com/pricing/self-managed/feature-comparison/), for information on what features are available at each tier. +- [GitLab pricing page](https://about.gitlab.com/pricing/#self-managed), for subscription information and a free trial. +- Our [product marketing page](https://about.gitlab.com/handbook/marketing/product-marketing/), for additional information including: + - How [different tiers are licensed](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers). + - The different [GitLab distributions](https://about.gitlab.com/handbook/marketing/product-marketing/#distributions). -Future purchases will use the information in this section. The email listed in this section is used for the Customers Portal login and for license related email communication. +#### Subscribing through GitLab self-managed -### Self-managed: Apply your license file +To subscribe to GitLab through a self-managed installation: + +1. [Install](https://about.gitlab.com/install/) GitLab. +1. Complete the installation with + [administration tasks](https://docs.gitlab.com/ee/administration/). +1. Select the **Starter**, **Premium**, or **Ultimate** self-managed plan + through the [GitLab Subscription Manager](https://customers.gitlab.com/). +1. Apply your license file. After purchase, a license file is sent to the email + address associated to the GitLab Subscription Manager account, + which needs to be + [uploaded to your GitLab instance](../user/admin_area/license.md#uploading-your-license). + +TIP: **Tip:** +If you are purchasing a subscription for an existing **Core** self-managed +instance, ensure you are purchasing enough seats to +[cover your users](../user/admin_area/index.md#administering-users). + +## Managing subscriptions + +You can view and manage subscriptions through our +[GitLab Subscription Manager](https://customers.gitlab.com/). -After purchase, the license file is sent to the email address tied to the Customers portal account, which needs to be [uploaded to the GitLab instance](../user/admin_area/license.md#uploading-your-license). +### View subscription and seats + +Visit the +[GitLab Subscription Manager](https://customers.gitlab.com/subscriptions) to +view and manage: + +- The subscriptions you have purchased. +- The number of seats associated with the subscription. +- Retrieve copies of invoices. +- Change the credit card on file. -### Link your GitLab.com account with your Customers Portal account +For more information, please see our: + +- [Subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/). +- [Pricing page](https://about.gitlab.com/pricing/), which includes information + on our [true-up pricing policy](https://about.gitlab.com/handbook/product/pricing/#true-up-pricing) + when adding more users other than at the time of purchase. NOTE: **Note:** -This is *required* for GitLab.com subscriptions. +The GitLab Subscription Manager account can have the same email address as your +GitLab.com account, but is a _separate_ login. If the two accounts are +linked together, you can use the **Or sign in with GitLab.com** +link underneath the **Sign In** button. + +### Change billing information + +To change billing information: -Once signed into the customers portal, if your account is not already linked, you should be prompted to link your account with a "Link my GitLab Account" button. +1. Log in to [GitLab Subscription Manager](https://customers.gitlab.com/customers/sign_in). +1. Go to the **My Account** page. +1. Make the required changes to the **Account Details** information. +1. Click **Update Account**. -You can also go to the [My Account](https://customers.gitlab.com/customers/edit) page to add or change the GitLab.com account link. +NOTE: **Note:** +Future purchases will use the information in this section. +The email listed in this section is used for the GitLab Subscription Manager +login and for license-related email communication. -### Change the linked GitLab.com account for your Customers Portal account +### Manage GitLab.com account -To change which GitLab.com account is associated with a Customers Portal account, please follow these steps: +This section provided information specific to managing subscriptions with a +GitLab.com account. -1. Log into the [Customers Portal](https://customers.gitlab.com/customers/sign_in). -1. In a separate browser tab, visit [GitLab.com](https://gitlab.com) to ensure you are not logged in, or if you are, log out. -1. Back on the Customers Portal page, click [My Account](https://customers.gitlab.com/customers/edit) in the top menu. -1. Under `Your GitLab.com account`, click the `Change linked account` button. -1. Have the user you want associated log in to their [GitLab.com](https://gitlab.com) account. +#### Change linked account -### GitLab.com: Associate your namespace with your subscription +To change the GitLab.com account associated with a GitLab Subscription Manager +account: -Once your GitLab.com account is linked, you can go to your [Subscriptions](https://customers.gitlab.com/subscriptions) page to choose or change the namespace your subscription applies to. +1. Log in to the + [GitLab Subscription Manager](https://customers.gitlab.com/customers/sign_in). +1. Go to [GitLab.com](https://gitlab.com) in a separate browser tab. Ensure you + are not logged in. +1. On the GitLab Subscription Manager page, click + [**My Account**](https://customers.gitlab.com/customers/edit) in the top menu. +1. Under **Your GitLab.com account**, click **Change linked account** button. +1. Log in to [GitLab.com](https://gitlab.com) account to link to. -Please note that you need to be a group owner to associate a group to your subscription. +#### Change associated namespace -### Confirm or upgrade your GitLab.com subscription details within GitLab +With a linked GitLab.com account, go to the +[**Subscriptions**](https://customers.gitlab.com/subscriptions) page to choose +or change the namespace your subscription applies to. + +NOTE: **Note:** +Please note that you need to be a group owner to associate a group to your +subscription. -To see the status of your GitLab.com subscription, you can click on the Billings -section of the relevant namespace: +### Confirm or upgrade your subscription -- For individuals, this is located at <https://gitlab.com/profile/billings> under - in your Settings, -- For groups, this is located under the group's Settings dropdown, under Billing. +To see the status of your GitLab.com subscription, you can click on the +**Billings** section of the relevant namespace: -For groups, you can see details of your subscription - including your current -plan - in the included table: +- For individuals: + 1. Go to **User Avatar > Settings**. + 1. Click **Billing**. +- For groups, go to the group's **Settings** dropdown, under **Billing**. - +The following table describes details of your subscription for groups: | Field | Description | | ------ | ------ | -| Seats in subscription | If this is a paid plan, this represents the number of seats you've paid to support in your group. | -| Seats currently in use | The number of active seats currently in use. | -| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | +| Seats in subscription | If this is a paid plan, represents the number of seats you've paid to support in your group. | +| Seats currently in use | Number of active seats currently in use. | +| Max seats used | Highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | | Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | -| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | -| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | +| Subscription start date | Date your subscription started. If this is for a Free plan, is the date you transitioned off your group's paid plan. | +| Subscription end date | Date your current subscription will end. Does not apply to Free plans. | -### Subscription changes and your data +#### Extra Shared Runners pipeline minutes -When your subscription or trial expires, GitLab does not delete your data, however, depending on the tier and feature, it may become inaccessible. Please note that some features may not behave as expected if a graceful fallback is not currently implemented, such as [environment specific variables not being passed](https://gitlab.com/gitlab-org/gitlab-ce/issues/52825). +If you're using GitLab.com, you can purchase additional CI minutes so your +pipelines will not be blocked after you have used all your CI minutes from your +main quota. Additional minutes: + +- Are only used once the shared quota included in your subscription runs out. +- Roll over month to month. + +In order to purchase additional minutes, you should follow these steps: + +1. Go to **Group > Settings > Pipelines quota**. Once you are on that page, click on **Buy additional minutes**. + +  + +1. Locate the subscription card that is linked to your group on GitLab.com, + click on **Buy more CI minutes**, and complete the details about the transaction. + +  + +1. Once we have processed your payment, the extra CI minutes + will be synced to your Group and you can visualize it from the + **Group > Settings > Pipelines quota** page: + +  + +Be aware that: + +1. If you have purchased extra CI minutes before the purchase of a paid plan, + we will calculate a pro-rated charge for your paid plan. That means you may + be charged for less than one year since your subscription was previously + created with the extra CI minutes. +1. Once the extra CI minutes has been assigned to a Group they cannot be transferred + to a different Group. +1. If you have some minutes used over your default quota, these minutes will + be deducted from your Additional Minutes quota immediately after your purchase of additional + minutes. + +##### What happens when my CI minutes run out + +When the CI minutes run out, an email is sent automatically to notify the owner(s) +of the group/namespace, including a link to [purchase more minutes](https://customers.gitlab.com/plans). + +If you are not the owner of the group, you will need to contact them to let them know they need to +[purchase more minutes](https://customers.gitlab.com/plans). + +## Subscription changes and your data + +When your subscription or trial expires, GitLab does not delete your data. + +However, depending on the tier and feature, your data may become inaccessible. + +Please note that some features may not behave as expected if a graceful +fallback is not currently implemented. For example, +[environment specific variables not being passed](https://gitlab.com/gitlab-org/gitlab-ce/issues/52825). If you renew or upgrade, your data will again be accessible. -For self-managed customers, there is a two-week grace period when your features will continue to work as-is, after which the entire instance will become read only. However, if you remove the license, you will immediately revert to Core features. +### Self-managed data + +For self-managed customers, there is a two-week grace period when your features +will continue to work as-is, after which the entire instance will become read +only. + +However, if you remove the license, you will immediately revert to Core +features. ## Need help? -[GitLab's Documentation](https://docs.gitlab.com/) offers a wide range of topics covering the use and administration of GitLab. +[GitLab's Documentation](https://docs.gitlab.com/) offers a wide range of +topics covering the use and administration of GitLab. -We also encourage all users to search our project trackers for known issues and existing feature requests in: +We also encourage all users to search our project trackers for known issues and +existing feature requests in: -- [GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce/issues/) for features included in all tiers, and -- [GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee/issues/) for paid-tier features. +- [GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce/issues/) for features + included in all tiers. +- [GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee/issues/) for paid-tier + features. -These issues are the best avenue for getting updates on specific product plans and for communicating directly with the relevant GitLab team members. +These issues are the best avenue for getting updates on specific product plans +and for communicating directly with the relevant GitLab team members. ### Contacting Support -Learn more about the tiers of [GitLab Support](https://about.gitlab.com/support/) or [submit a request via the Support Portal](https://support.gitlab.com/hc/en-us/requests/new). +Learn more about: + +- The tiers of [GitLab Support](https://about.gitlab.com/support/). +- [Submit a request via the Support Portal](https://support.gitlab.com/hc/en-us/requests/new). <!-- ## Troubleshooting diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index a46f7d30892..1e9eb15533a 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -330,6 +330,7 @@ If the user is blocked via LDAP, `state` will be `ldap_blocked`. "user_id": 41 } ``` + **Group Member Removed:** ```json @@ -643,6 +644,11 @@ X-Gitlab-Event: System Hook } ``` +## Local requests in system hooks + +[Requests to local network by system hooks](../security/webhooks.md) can be allowed +or blocked by an administrator. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/tools/email.md b/doc/tools/email.md index 72a5d094bc9..3088b0b63e7 100644 --- a/doc/tools/email.md +++ b/doc/tools/email.md @@ -19,12 +19,12 @@ at their primary email address. 1. Go to the admin area using the wrench icon in the top right corner and navigate to **Overview > Users > Send email to users**. -  +  1. Compose an email and choose where it will be sent (all users or users of a chosen group or project): -  +  ## Unsubscribing from emails diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 8b4a2f1630b..ad196e27f53 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -23,7 +23,7 @@ This page gathers all the resources for the topic **Authentication** within GitL - [How to Configure LDAP with GitLab CE](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md) - [How to Configure LDAP with GitLab EE](../../administration/auth/how_to_configure_ldap_gitlab_ee/index.md) **(STARTER)** - [Feature Highlight: LDAP Integration](https://about.gitlab.com/2014/07/10/feature-highlight-ldap-sync/) - - [Debugging LDAP](https://about.gitlab.com/handbook/support/workflows/support-engineering/ldap/debugging_ldap.html) + - [Debugging LDAP](https://about.gitlab.com/handbook/support/workflows/debugging_ldap.html) - **Integrations:** - [OmniAuth](../../integration/omniauth.md) - [Authentiq OmniAuth Provider](../../administration/auth/authentiq.md#authentiq-omniauth-provider) diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 503ad784a77..6021e8cff1d 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -1,6 +1,7 @@ # Auto DevOps -> [Introduced][ce-37115] in GitLab 10.0. Generally available on GitLab 11.0. +> - [Introduced][ce-37115] in GitLab 10.0. +> - Generally available on GitLab 11.0. Auto DevOps provides pre-defined CI/CD configuration which allows you to automatically detect, build, test, deploy, and monitor your applications. Leveraging CI/CD best practices and tools, Auto DevOps aims @@ -43,16 +44,16 @@ platform or a Platform as a Service (PaaS). It takes inspiration from the innovative work done by [Heroku](https://www.heroku.com/) and goes beyond it in multiple ways: -1. Auto DevOps works with any Kubernetes cluster; you're not limited to running - on GitLab's infrastructure. (Note that many features also work without Kubernetes.) -1. There is no additional cost (no markup on the infrastructure costs), and you - can use a self-hosted Kubernetes cluster or Containers as a Service on any - public cloud (for example, [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)). -1. Auto DevOps has more features including security testing, performance testing, - and code quality testing. -1. Auto DevOps offers an incremental graduation path. If you need advanced customizations, - you can start modifying the templates without having to start over on a - completely different platform. Review the [customizing](#customizing) section for more information. +- Auto DevOps works with any Kubernetes cluster; you're not limited to running + on GitLab's infrastructure. (Note that many features also work without Kubernetes). +- There is no additional cost (no markup on the infrastructure costs), and you + can use a self-hosted Kubernetes cluster or Containers as a Service on any + public cloud (for example, [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/)). +- Auto DevOps has more features including security testing, performance testing, + and code quality testing. +- Auto DevOps offers an incremental graduation path. If you need advanced customizations, + you can start modifying the templates without having to start over on a + completely different platform. Review the [customizing](#customizing) section for more information. ## Features @@ -64,7 +65,7 @@ project in a simple and automatic way: 1. [Auto Code Quality](#auto-code-quality-starter) **(STARTER)** 1. [Auto SAST (Static Application Security Testing)](#auto-sast-ultimate) **(ULTIMATE)** 1. [Auto Dependency Scanning](#auto-dependency-scanning-ultimate) **(ULTIMATE)** -1. [Auto License Management](#auto-license-management-ultimate) **(ULTIMATE)** +1. [Auto License Compliance](#auto-license-compliance-ultimate) **(ULTIMATE)** 1. [Auto Container Scanning](#auto-container-scanning-ultimate) **(ULTIMATE)** 1. [Auto Review Apps](#auto-review-apps) 1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast-ultimate) **(ULTIMATE)** @@ -86,42 +87,45 @@ Auto DevOps provides great defaults for all the stages; you can, however, For an overview on the creation of Auto DevOps, read the blog post [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/). +NOTE: **Note** +Kubernetes clusters can [be used without](../../user/project/clusters/index.md) +Auto DevOps. + ## Requirements To make full use of Auto DevOps, you will need: -1. **GitLab Runner** (needed for all stages) - Your Runner needs to be - configured to be able to run Docker. Generally this means using the - [Docker](https://docs.gitlab.com/runner/executors/docker.html) or [Kubernetes - executor](https://docs.gitlab.com/runner/executors/kubernetes.html), with - [privileged mode enabled](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode). - The Runners do not need to be installed in the Kubernetes cluster, but the - Kubernetes executor is easy to use and is automatically autoscaling. - Docker-based Runners can be configured to autoscale as well, using [Docker - Machine](https://docs.gitlab.com/runner/install/autoscaling.html). Runners - should be registered as [shared Runners](../../ci/runners/README.md#registering-a-shared-runner) - for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner) - that are assigned to specific projects. -1. **Base domain** (needed for Auto Review Apps and Auto Deploy) - You will need - a domain configured with wildcard DNS which is going to be used by all of your - Auto DevOps applications. [Read the specifics](#auto-devops-base-domain). -1. **Kubernetes** (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring) - - To enable deployments, you will need Kubernetes 1.5+. You need a [Kubernetes cluster][kubernetes-clusters] - for the project, or a Kubernetes [default service template](../../user/project/integrations/services_templates.md) - for the entire GitLab installation. - 1. **A load balancer** - You can use NGINX ingress by deploying it to your - Kubernetes cluster using the - [`nginx-ingress`](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress) - Helm chart. -1. **Prometheus** (needed for Auto Monitoring) - To enable Auto Monitoring, you - will need Prometheus installed somewhere (inside or outside your cluster) and - configured to scrape your Kubernetes cluster. To get response metrics - (in addition to system metrics), you need to - [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). - The [Prometheus service](../../user/project/integrations/prometheus.md) - integration needs to be enabled for the project, or enabled as a - [default service template](../../user/project/integrations/services_templates.md) - for the entire GitLab installation. +- **GitLab Runner** (needed for all stages) - Your Runner needs to be + configured to be able to run Docker. Generally this means using the + [Docker](https://docs.gitlab.com/runner/executors/docker.html) or [Kubernetes + executor](https://docs.gitlab.com/runner/executors/kubernetes.html), with + [privileged mode enabled](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode). + The Runners do not need to be installed in the Kubernetes cluster, but the + Kubernetes executor is easy to use and is automatically autoscaling. + Docker-based Runners can be configured to autoscale as well, using [Docker + Machine](https://docs.gitlab.com/runner/install/autoscaling.html). Runners + should be registered as [shared Runners](../../ci/runners/README.md#registering-a-shared-runner) + for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner) + that are assigned to specific projects. +- **Base domain** (needed for Auto Review Apps and Auto Deploy) - You will need + a domain configured with wildcard DNS which is going to be used by all of your + Auto DevOps applications. [Read the specifics](#auto-devops-base-domain). +- **Kubernetes** (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring) - + To enable deployments, you will need Kubernetes 1.5+. You will need a [Kubernetes cluster][kubernetes-clusters] + for the project. + - **A load balancer** - You can use NGINX ingress by deploying it to your + Kubernetes cluster using the + [`nginx-ingress`](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress) + Helm chart. +- **Prometheus** (needed for Auto Monitoring) - To enable Auto Monitoring, you + will need Prometheus installed somewhere (inside or outside your cluster) and + configured to scrape your Kubernetes cluster. To get response metrics + (in addition to system metrics), you need to + [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). + The [Prometheus service](../../user/project/integrations/prometheus.md) + integration needs to be enabled for the project, or enabled as a + [default service template](../../user/project/integrations/services_templates.md) + for the entire GitLab installation. If you do not have Kubernetes or Prometheus installed, then Auto Review Apps, Auto Deploy, and Auto Monitoring will be silently skipped. @@ -147,7 +151,7 @@ NOTE: **Note** A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: -``` +```text *.example.com 3600 A 1.2.3.4 ``` @@ -168,7 +172,6 @@ From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`. Support for `AUTO_DEVOPS_DOMAIN` was [removed in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959). - ## Using multiple Kubernetes clusters **(PREMIUM)** When using Auto DevOps, you may want to deploy different environments to @@ -186,23 +189,23 @@ Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so except for the environment scope, they would also need to have a different domain they would be deployed to. This is why you need to define a separate `KUBE_INGRESS_BASE_DOMAIN` variable for all the above -[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium). +[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables). The following table is an example of how the three different clusters would be configured. | Cluster name | Cluster environment scope | `KUBE_INGRESS_BASE_DOMAIN` variable value | Variable environment scope | Notes | -| ------------ | -------------- | ----------------------------- | ------------- | ------ | -| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. | -| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). | -| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production-premium). | +|--------------|---------------------------|-------------------------------------------|----------------------------|---| +| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. | +| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). | +| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production-premium). | To add a different cluster for each environment: 1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters with their respective environment scope as described from the table above. -  +  1. After the clusters are created, navigate to each one and install Helm Tiller and Ingress. Wait for the Ingress IP address to be assigned. @@ -223,9 +226,9 @@ full use of Auto DevOps are available. If this is your fist time, we recommend y [quick start guide](quick_start_guide.md). GitLab.com users can enable/disable Auto DevOps at the project-level only. Self-managed users -can enable/disable Auto DevOps at either the project-level or instance-level. +can enable/disable Auto DevOps at the project-level, group-level or instance-level. -### Enabling/disabling Auto DevOps at the instance-level (Administrators only) +### At the instance level (Administrators only) Even when disabled at the instance level, group owners and project maintainers can still enable Auto DevOps at the group and project level, respectively. @@ -235,7 +238,7 @@ Auto DevOps at the group and project level, respectively. 1. If enabling, optionally set up the Auto DevOps [base domain](#auto-devops-base-domain) which will be used for Auto Deploy and Auto Review Apps. 1. Click **Save changes** for the changes to take effect. -### Enabling/disabling Auto DevOps at the group-level +### At the group level > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/52447) in GitLab 11.10. @@ -251,7 +254,7 @@ When enabling or disabling Auto DevOps at group-level, group configuration will the subgroups and projects inside that group, unless Auto DevOps is specifically enabled or disabled on the subgroup or project. -### Enabling/disabling Auto DevOps at the project-level +### At the project level If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one exists, remove it. @@ -264,7 +267,7 @@ If enabling, check that your project doesn't have a `.gitlab-ci.yml`, or if one When the feature has been enabled, an Auto DevOps pipeline is triggered on the default branch. -### Feature flag to enable for a percentage of projects +### Enable for a percentage of projects There is also a feature flag to enable Auto DevOps by default to your chosen percentage of projects. @@ -372,7 +375,7 @@ Any differences between the source and target branches are also Static Application Security Testing (SAST) uses the [SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) to run static analysis on the current code and checks for potential security issues. The -the Auto SAST stage will be skipped on licenses other than Ultimate and requires GitLab Runner 11.5 or above. +Auto SAST stage will be skipped on licenses other than Ultimate and requires GitLab Runner 11.5 or above. Once the report is created, it's uploaded as an artifact which you can later download and check out. @@ -397,13 +400,13 @@ check out. Any security warnings are also shown in the merge request widget. Read more about [Dependency Scanning](../../user/application_security/dependency_scanning/index.md). -### Auto License Management **(ULTIMATE)** +### Auto License Compliance **(ULTIMATE)** > Introduced in [GitLab Ultimate][ee] 11.0. -License Management uses the -[License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) -to search the project dependencies for their license. The Auto License Management stage +License Compliance uses the +[License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +to search the project dependencies for their license. The Auto License Compliance stage will be skipped on licenses other than Ultimate. Once the @@ -411,7 +414,7 @@ report is created, it's uploaded as an artifact which you can later download and check out. Any licenses are also shown in the merge request widget. Read more how -[License Management works](../../user/application_security/license_management/index.md). +[License Compliance works](../../user/application_security/license_management/index.md). ### Auto Container Scanning **(ULTIMATE)** @@ -489,7 +492,7 @@ Any security warnings are also shown in the merge request widget. Read how Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) to measure the performance of a web page. A JSON report is created and uploaded as an artifact, which includes the overall performance score for each page. By default, the root page of Review and Production environments will be tested. If you would like to add additional URL's to test, simply add the paths to a file named `.gitlab-urls.txt` in the root directory, one per line. For example: -``` +```text / /features /direction @@ -580,6 +583,55 @@ Unless you have a `Dockerfile` in your repo, your image is built with Herokuish, and you must prefix commands run in these images with `/bin/herokuish procfile exec` to replicate the environment where your application will run. +#### Workers + +Some web applications need to run extra deployments for "worker processes". For +example, it is common in a Rails application to have a separate worker process +to run background tasks like sending emails. + +The [default Helm chart](https://gitlab.com/gitlab-org/charts/auto-deploy-app) +used in Auto Deploy [has support for running worker +processes](https://gitlab.com/gitlab-org/charts/auto-deploy-app/merge_requests/9). + +In order to run a worker, you'll need to ensure that it is able to respond to +the standard health checks, which expect a successful HTTP response on port +`5000`. For [Sidekiq](https://github.com/mperham/sidekiq), you could make use of +the [`sidekiq_alive` gem](https://rubygems.org/gems/sidekiq_alive) to do this. + +In order to work with Sidekiq, you'll also need to ensure your deployments have +access to a Redis instance. Auto DevOps won't deploy this for you so you'll +need to: + +- Maintain your own Redis instance. +- Set a CI variable `K8S_SECRET_REDIS_URL`, which the URL of this instance to + ensure it's passed into your deployments. + +Once you have configured your worker to respond to health checks, you will +need to configure a CI variable `HELM_UPGRADE_EXTRA_ARGS` with the value +`--values helm-values.yaml`. + +Then you can, for example, run a Sidekiq worker for your Rails application +by adding a file named `helm-values.yaml` to your repository with the following +content: + +```yml +workers: + sidekiq: + replicaCount: 1 + command: + - /bin/herokuish + - procfile + - exec + - sidekiq + preStopCommand: + - /bin/herokuish + - procfile + - exec + - sidekiqctl + - quiet + terminationGracePeriodSeconds: 60 +``` + ### Auto Monitoring See the [requirements](#requirements) for Auto Monitoring to enable this stage. @@ -658,17 +710,13 @@ repo or by specifying a project variable: You can also make use of the `HELM_UPGRADE_EXTRA_ARGS` environment variable to override the default values in the `values.yaml` file in the [default Helm chart](https://gitlab.com/gitlab-org/charts/auto-deploy-app). To apply your own `values.yaml` file to all Helm upgrade commands in Auto Deploy set `HELM_UPGRADE_EXTRA_ARGS` to `--values my-values.yaml`. -### Custom Helm chart per environment **(PREMIUM)** +### Custom Helm chart per environment You can specify the use of a custom Helm chart per environment by scoping the environment variable -to the desired environment. See [Limiting environment scopes of variables](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium). +to the desired environment. See [Limiting environment scopes of variables](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables). ### Customizing `.gitlab-ci.yml` -Everything about Auto DevOps is customizable since the [Auto DevOps template] -is just an example of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) and uses -only features that are available to any `.gitlab-ci.yml`. - Auto DevOps is completely customizable because the [Auto DevOps template]: - Is just an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) file. @@ -725,46 +773,34 @@ The following variables can be used for setting up the Auto DevOps domain, providing a custom Helm chart, or scaling your application. PostgreSQL can also be customized, and you can easily use a [custom buildpack](#custom-buildpacks). -| **Variable** | **Description** | -| ------------ | --------------- | -| `AUTO_DEVOPS_CHART` | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/charts/auto-deploy-app). | -| `AUTO_DEVOPS_CHART_REPOSITORY` | The Helm Chart repository used to search for charts; defaults to `https://charts.gitlab.io`. | -| `AUTO_DEVOPS_CHART_REPOSITORY_NAME` | From Gitlab 11.11, this variable can be used to set the name of the helm repository; defaults to "gitlab" | -| `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From Gitlab 11.11, this variable can be used to set a username to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD) | -| `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From Gitlab 11.11, this variable can be used to set a password to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_USERNAME) | -| `REPLICAS` | The number of replicas to deploy; defaults to 1. | -| `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | -| `CANARY_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md); defaults to 1 | -| `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1 | -| `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. | -| `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`. | -| `POSTGRES_ENABLED` | Whether PostgreSQL is enabled; defaults to `"true"`. Set to `false` to disable the automatic deployment of PostgreSQL. | -| `POSTGRES_USER` | The PostgreSQL user; defaults to `user`. Set it to use a custom username. | -| `POSTGRES_PASSWORD` | The PostgreSQL password; defaults to `testing-password`. Set it to use a custom password. | -| `POSTGRES_DB` | The PostgreSQL database name; defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-environment-variables). Set it to use a custom database name. | -| `POSTGRES_VERSION` | Tag for the [`postgres` Docker image](https://hub.docker.com/_/postgres) to use. Defaults to `9.6.2`. | -| `BUILDPACK_URL` | The buildpack's full URL. It can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`, for example `https://github.com/heroku/heroku-buildpack-ruby.git#v142` | -| `SAST_CONFIDENCE_LEVEL` | The minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High; defaults to `3`.| -| `DEP_SCAN_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled; defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).| -| `DB_INITIALIZE` | From GitLab 11.4, this variable can be used to specify the command to run to initialize the application's PostgreSQL database. It runs inside the application pod. | -| `DB_MIGRATE` | From GitLab 11.4, this variable can be used to specify the command to run to migrate the application's PostgreSQL database. It runs inside the application pod. | -| `STAGING_ENABLED` | From GitLab 10.8, this variable can be used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). | -| `CANARY_ENABLED` | From GitLab 11.0, this variable can be used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments-premium). | -| `INCREMENTAL_ROLLOUT_MODE`| From GitLab 11.4, this variable, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production-premium) of your application for the production environment.<br/>Set to: <ul><li>`manual`, for manual deployment jobs.</li><li>`timed`, for automatic rollout deployments with a 5 minute delay each one.</li></ul> | -| `TEST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `test` job. If the variable is present, the job will not be created. | -| `CODE_QUALITY_DISABLED` | From GitLab 11.0, this variable can be used to disable the `codequality` job. If the variable is present, the job will not be created. | -| `LICENSE_MANAGEMENT_DISABLED` | From GitLab 11.0, this variable can be used to disable the `license_management` job. If the variable is present, the job will not be created. | -| `SAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. | -| `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dependency_scanning` job. If the variable is present, the job will not be created. | -| `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast:container` job. If the variable is present, the job will not be created. | -| `REVIEW_DISABLED` | From GitLab 11.0, this variable can be used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. | -| `DAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. | -| `PERFORMANCE_DISABLED` | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. | -| `K8S_SECRET_*` | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) will be made available by Auto DevOps as environment variables to the deployed application. | -| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/index.md#base-domain) for more information. | -| `ROLLOUT_RESOURCE_TYPE` | From GitLab 11.9, this variable allows specification of the resource type being deployed when using a custom helm chart. Default value is `deployment`. | -| `ROLLOUT_STATUS_DISABLED` | From GitLab 12.0, this variable allows to disable rollout status check because it doesn't support all resource types, for example, `cronjob`. | -| `HELM_UPGRADE_EXTRA_ARGS` | From GitLab 11.11, this variable allows extra arguments in `helm` commands when deploying the application. Note that using quotes will not prevent word splitting. **Tip:** you can use this variable to [customize the Auto Deploy helm chart](https://docs.gitlab.com/ee/topics/autodevops/index.html#custom-helm-chart) by applying custom override values with `--values my-values.yaml`. | +#### Build and deployment + +The following table lists variables related to building and deploying +applications. + +| **Variable** | **Description** | +|-----------------------------------------|------------------------------------| +| `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. | +| `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`. | +| `AUTO_DEVOPS_CHART` | Helm Chart used to deploy your apps. Defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/charts/auto-deploy-app). | +| `AUTO_DEVOPS_CHART_REPOSITORY` | Helm Chart repository used to search for charts. Defaults to `https://charts.gitlab.io`. | +| `AUTO_DEVOPS_CHART_REPOSITORY_NAME` | From Gitlab 11.11, used to set the name of the helm repository. Defaults to `gitlab`. | +| `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From Gitlab 11.11, used to set a username to connect to the helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD`. | +| `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From Gitlab 11.11, used to set a password to connect to the helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. | +| `BUILDPACK_URL` | Buildpack's full URL. Can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`. For example `https://github.com/heroku/heroku-buildpack-ruby.git#v142`. | +| `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments-premium). | +| `CANARY_PRODUCTION_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. Takes precedence over `CANARY_REPLICAS`. Defaults to 1. | +| `CANARY_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md). Defaults to 1. | +| `HELM_RELEASE_NAME` | From GitLab 12.1, allows the `helm` release name to be overridden. Can be used to assign unique release names when deploying multiple projects to a single namespace. | +| `HELM_UPGRADE_EXTRA_ARGS` | From GitLab 11.11, allows extra arguments in `helm` commands when deploying the application. Note that using quotes will not prevent word splitting. **Tip:** you can use this variable to [customize the Auto Deploy helm chart](#custom-helm-chart) by applying custom override values with `--values my-values.yaml`. | +| `INCREMENTAL_ROLLOUT_MODE` | From GitLab 11.4, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production-premium) of your application for the production environment. Set to `manual` for manual deployment jobs or `timed` for automatic rollout deployments with a 5 minute delay each one. | +| `K8S_SECRET_*` | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) will be made available by Auto DevOps as environment variables to the deployed application. | +| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/index.md#base-domain) for more information. | +| `PRODUCTION_REPLICAS` | Number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | +| `REPLICAS` | Number of replicas to deploy. Defaults to 1. | +| `ROLLOUT_RESOURCE_TYPE` | From GitLab 11.9, allows specification of the resource type being deployed when using a custom helm chart. Default value is `deployment`. | +| `ROLLOUT_STATUS_DISABLED` | From GitLab 12.0, used to disable rollout status check because it doesn't support all resource types, for example, `cronjob`. | +| `STAGING_ENABLED` | From GitLab 10.8, used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). | TIP: **Tip:** Set up the replica variables using a @@ -776,6 +812,42 @@ You should *not* scale your application using Kubernetes directly. This can cause confusion with Helm not detecting the change, and subsequent deploys with Auto DevOps can undo your changes. +#### Database + +The following table lists variables related to the database. + +| **Variable** | **Description** | +| `DB_INITIALIZE` | From GitLab 11.4, used to specify the command to run to initialize the application's PostgreSQL database. Runs inside the application pod. | +| `DB_MIGRATE` | From GitLab 11.4, used to specify the command to run to migrate the application's PostgreSQL database. Runs inside the application pod. | +| `POSTGRES_ENABLED` | Whether PostgreSQL is enabled. Defaults to `"true"`. Set to `false` to disable the automatic deployment of PostgreSQL. | +| `POSTGRES_USER` | The PostgreSQL user. Defaults to `user`. Set it to use a custom username. | +| `POSTGRES_PASSWORD` | The PostgreSQL password. Defaults to `testing-password`. Set it to use a custom password. | +| `POSTGRES_DB` | The PostgreSQL database name. Defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-environment-variables). Set it to use a custom database name. | +| `POSTGRES_VERSION` | Tag for the [`postgres` Docker image](https://hub.docker.com/_/postgres) to use. Defaults to `9.6.2`. | + +#### Security tools + +The following table lists variables related to security tools. + +| **Variable** | **Description** | +| `SAST_CONFIDENCE_LEVEL` | Minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High. Defaults to `3`. | +| `DS_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled. Defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](../../user/application_security/dependency_scanning/index.md#remote-checks). | + +#### Disable jobs + +The following table lists variables used to disable jobs. + +| **Variable** | **Description** | +| `CODE_QUALITY_DISABLED` | From GitLab 11.0, used to disable the `codequality` job. If the variable is present, the job will not be created. | +| `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0, used to disable the `sast:container` job. If the variable is present, the job will not be created. | +| `DAST_DISABLED` | From GitLab 11.0, used to disable the `dast` job. If the variable is present, the job will not be created. | +| `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0, used to disable the `dependency_scanning` job. If the variable is present, the job will not be created. | +| `LICENSE_MANAGEMENT_DISABLED` | From GitLab 11.0, used to disable the `license_management` job. If the variable is present, the job will not be created. | +| `PERFORMANCE_DISABLED` | From GitLab 11.0, used to disable the `performance` job. If the variable is present, the job will not be created. | +| `REVIEW_DISABLED` | From GitLab 11.0, used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. | +| `SAST_DISABLED` | From GitLab 11.0, used to disable the `sast` job. If the variable is present, the job will not be created. | +| `TEST_DISABLED` | From GitLab 11.0, used to disable the `test` job. If the variable is present, the job will not be created. | + #### Application secret variables > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/49056) in GitLab 11.7. @@ -790,11 +862,11 @@ To configure your application variables: 1. Go to your project's **Settings > CI/CD**, then expand the section called **Variables**. -2. Create a CI Variable, ensuring the key is prefixed with +1. Create a CI Variable, ensuring the key is prefixed with `K8S_SECRET_`. For example, you can create a variable with key -`K8S_SECRET_RAILS_MASTER_KEY`. + `K8S_SECRET_RAILS_MASTER_KEY`. -3. Run an Auto Devops pipeline either by manually creating a new +1. Run an Auto Devops pipeline either by manually creating a new pipeline or by pushing a code change to GitLab. Auto DevOps pipelines will take your application secret variables to @@ -993,10 +1065,10 @@ Everything behaves the same way, except: - It's enabled by setting the `INCREMENTAL_ROLLOUT_MODE` variable to `timed`. - Instead of the standard `production` job, the following jobs with a 5 minute delay between each are created: - 1. `timed rollout 10%` - 1. `timed rollout 25%` - 1. `timed rollout 50%` - 1. `timed rollout 100%` + 1. `timed rollout 10%` + 1. `timed rollout 25%` + 1. `timed rollout 50%` + 1. `timed rollout 100%` ## Currently supported languages @@ -1009,7 +1081,7 @@ multi-buildpack does not. As of GitLab 10.0, the supported buildpacks are: -``` +```text - heroku-buildpack-multi v1.0.0 - heroku-buildpack-ruby v168 - heroku-buildpack-nodejs v99 @@ -1045,6 +1117,12 @@ Container Registry. **Restarting a pod, scaling a service, or other actions whic require on-going access to the registry may fail**. On-going secure access is planned for a subsequent release. +### Private registry support + +There is no documented way of using private container registry with Auto DevOps. +We strongly advise using GitLab Container Registry with Auto DevOps in order to +simplify configuration and prevent any unforeseen issues. + ## Troubleshooting - Auto Build and Auto Test may fail in detecting your language/framework. There @@ -1057,7 +1135,7 @@ planned for a subsequent release. case, you may need to customize your `.gitlab-ci.yml` with your test commands. - Auto Deploy will fail if GitLab can not create a Kubernetes namespace and service account for your project. For help debugging this issue, see - [Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting-failed-deployment-jobs). + [Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting). ### Disable the banner instance wide diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index c1771a57da0..35a5aff6a60 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -38,13 +38,13 @@ those projects provide a barebones application built on some well-known framewor Rails, Spring, or NodeJS Express project. For this example, we'll use the Ruby on Rails template. -  +  1. Give your project a name, optionally a description, and make it public so that you can take advantage of the features available in the [GitLab Gold plan](https://about.gitlab.com/pricing/#gitlab-com). -  +  1. Click **Create project**. @@ -56,20 +56,20 @@ under which this application will be deployed. 1. On the project's landing page, click the button labeled **Add Kubernetes cluster** (note that this option is also available when you navigate to **Operations > Kubernetes**). -  +  1. Choose **Create on Google Kubernetes Engine**. -  +  1. Sign in with Google. -  +  1. Connect with your Google account and press **Allow** when asked (this will be shown only the first time you connect GitLab with your Google account). -  +  1. The last step is to fill in the cluster details. Give it a name, leave the environment scope as is, and choose the GCP project under which the cluster @@ -80,7 +80,7 @@ under which this application will be deployed. cluster will be created, enter the number of nodes you want it to have, and finally choose their [machine type](https://cloud.google.com/compute/docs/machine-types). -  +  1. Once ready, click **Create Kubernetes cluster**. @@ -133,7 +133,7 @@ Now that the Kubernetes cluster is set up and ready, let's enable Auto DevOps. successfully runs on the `master` branch. 1. Click **Save changes**. -  +  Once you complete all the above and save your changes, a new pipeline is automatically created. To view the pipeline, go to **CI/CD > Pipelines**. @@ -167,7 +167,7 @@ In the **test** stage, GitLab runs various checks on the application: - The `sast` job runs static analysis on the current code to check for potential security issues and is allowed to fail([Auto SAST](index.md#auto-sast-ultimate)) **(ULTIMATE)** - The `license_management` job searches the application's dependencies to determine each of their - licenses and is allowed to fail ([Auto License Management](index.md#auto-license-management-ultimate)) **(ULTIMATE)** + licenses and is allowed to fail ([Auto License Compliance](index.md#auto-license-compliance-ultimate)) **(ULTIMATE)** NOTE: **Note:** As you might have noticed, all jobs except `test` are allowed to fail in the @@ -201,7 +201,7 @@ applications. In the rightmost column for the production environment, you can ma Prometheus collects data about the Kubernetes cluster and how the application affects it (in terms of memory/CPU usage, latency, etc.). -  +  - The third icon is the [web terminal](../../ci/environments.md#web-terminals) and it will open a terminal session right inside the container where the diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index cdcd8215b23..6a539b526f3 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -48,6 +48,7 @@ The following are resources about version control concepts: The following resources may help you become more efficient at using Git: +- [Useful Git commands](useful_git_commands.md) collected by the GitLab support team. - [Git Tips & Tricks](https://about.gitlab.com/2016/12/08/git-tips-and-tricks/) - [Eight Tips to help you work better with Git](https://about.gitlab.com/2015/02/19/8-tips-to-help-you-work-better-with-git/) @@ -71,6 +72,7 @@ The following are advanced topics for those who want to get the most out of Git: - [Custom Git Hooks](../../administration/custom_hooks.md) - [Git Attributes](../../user/project/git_attributes.md) - Git Submodules: [Using Git submodules with GitLab CI](../../ci/git_submodules.md#using-git-submodules-with-gitlab-ci) +- [Partial Clone](partial_clone.md) ## API @@ -82,6 +84,8 @@ Git-related queries from GitLab. The following relate to Git Large File Storage: - [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) -- [GitLab Git LFS documentation](../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- [Migrate an existing Git repo with Git LFS](migrate_to_git_lfs/index.md) +- [GitLab Git LFS user documentation](../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- [GitLab Git LFS admin documentation](../../workflow/lfs/lfs_administration.md) - [Git-Annex to Git-LFS migration guide](../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md) - [Towards a production quality open source Git LFS server](https://about.gitlab.com/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) diff --git a/doc/topics/git/migrate_to_git_lfs/index.md b/doc/topics/git/migrate_to_git_lfs/index.md new file mode 100644 index 00000000000..c879e404997 --- /dev/null +++ b/doc/topics/git/migrate_to_git_lfs/index.md @@ -0,0 +1,174 @@ +--- +type: tutorial, concepts +description: "How to migrate an existing Git repository to Git LFS with BFG." +last_updated: 2019-07-11 +--- + +# Migrate a Git repo into Git LFS with BFG + +Using Git LFS can help you to reduce the size of your Git +repository and improve its performance. + +However, simply adding the +large files that are already in your repository to Git LFS, +will not actually reduce the size of your repository because +the files are still referenced by previous commits. + +Through the method described on this document, first migrate +to Git LFS with [BFG](https://rtyley.github.io/bfg-repo-cleaner/) +through a mirror repo, then clean up the repository's history, +and lastly create LFS tracking rules to prevent new binary files +from being added. + +This tutorial was inspired by the guide +[Use BFG to migrate a repo to Git LFS](https://confluence.atlassian.com/bitbucket/use-bfg-to-migrate-a-repo-to-git-lfs-834233484.html). +For more information on Git LFS, see the [references](#references) +below. + +CAUTION: **Warning:** +The method described on this guide rewrites Git history. Make +sure to back up your repo before beginning and use it at your +own risk. + +## Requirements + +Before beginning, make sure: + +- You have enough LFS storage for the files you want to convert. + Storage is required for the entire history of all files. +- All the team members you share the repository with have pushed all changes. + Branches based on the repository before applying this method cannot be merged. + Branches based on the repo before applying this method cannot be merged. + +To follow this tutorial, you'll need: + +- Maintainer permissions to the existing Git repository + you'd like to migrate to LFS with access through the command line. +- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) + and [Java Runtime Environment](https://www.java.com/en/download/manual.jsp) + (Java 7 or above) installed locally. +- BFG installed locally: + + ```bash + brew install bfg + ``` + +- Git LFS installed locally: + + ```bash + brew install git-lfs + ``` + +NOTE: **Note:** +This guide was tested on macOS Mojave. + +## Steps + +Consider an example upstream project, `git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git`. + +1. Back up your repository: + + Create a copy of your repository so that you can + recover it in case something goes wrong. + +1. Clone `--mirror` the repo: + + Cloning with the mirror flag will create a bare repository. + This ensures you get all the branches within the repo. + + It creates a directory called `<repo-name>.git` + (in our example, `test-git-lfs-repo-migration.git`), + mirroring the upstream project: + + ```bash + git clone --mirror git@gitlab.com:gitlab-tests/test-git-lfs-repo-migration.git + ``` + +1. Convert the Git history with BFG: + + ```bash + bfg --convert-to-git-lfs "*.{png,mp4,jpg,gif}" --no-blob-protection test-git-lfs-repo-migration.git + ``` + + It is scanning all the history, and looking for any files with + that extension, and then converting them to an LFS pointer. + +1. Clean up the repository: + + ```bash + # cd path/to/mirror/repo: + cd test-git-lfs-repo-migration.git + # clean up the repo: + git reflog expire --expire=now --all && git gc --prune=now --aggressive + ``` + + You can also take a look on how to further [clean the repo](../../../user/project/repository/reducing_the_repo_size_using_git.md), + but it's not necessary for the purposes of this guide. + +1. Install Git LFS in the mirror repository: + + ```bash + git lfs install + ``` + +1. [Unprotect the default branch](../../../user/project/protected_branches.md), + so that we can force-push the rewritten repository: + + 1. Navigate to your project's **Settings > Repository** and + expand **Protected Branches**. + 1. Scroll down to locate the protected branches and click + **Unprotect** the default branch. + +1. Force-push to GitLab: + + ```bash + git push --force + ``` + +1. Track the files you want with LFS: + + ```bash + # cd path/to/upstream/repo: + cd test-git-lfs-repo-migration + # You may need to reset your local copy with upstream's `master` after force-pushing from the mirror: + git reset --hard origin/master + # Track the files with LFS: + git lfs track "*.gif" "*.png" "*.jpg" "*.psd" "*.mp4" ".gitattributes" "img/" + ``` + + Now all existing the files you converted, as well as the new + ones you add, will be properly tracked with LFS. + +1. [Re-protect the default branch](../../../user/project/protected_branches.md): + + 1. Navigate to your project's **Settings > Repository** and + expand **Protected Branches**. + 1. Select the default branch from the **Branch** dropdown menu, + and set up the + **Allowed to push** and **Allowed to merge** rules. + 1. Click **Protect**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + +## References + +- [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) +- [Migrate from Git Annex to Git LFS](../../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md) +- [GitLab's Git LFS user documentation](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- [GitLab's Git LFS administrator documentation](../../../workflow/lfs/lfs_administration.md) +- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs) + +<!-- +Test project: +https://gitlab.com/gitlab-tests/test-git-lfs-repo-migration +--> diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/index.md b/doc/topics/git/numerous_undo_possibilities_in_git/index.md index 84201e11831..5cae532bf54 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/index.md +++ b/doc/topics/git/numerous_undo_possibilities_in_git/index.md @@ -110,21 +110,21 @@ At this point there are 3 options to undo the local changes you have: - Discard all local changes, but save them for possible re-use [later](#quickly-save-local-changes): - ```shell - git stash - ``` + ```shell + git stash + ``` - Discarding local changes (permanently) to a file: - ```shell - git checkout -- <file> - ``` + ```shell + git checkout -- <file> + ``` - Discard all local changes to all files permanently: - ```shell - git reset --hard - ``` + ```shell + git reset --hard + ``` Before executing `git reset --hard`, keep in mind that there is also a way to just temporary store the changes without committing them using `git stash`. @@ -182,27 +182,27 @@ Now you have 4 options to undo your changes: - Unstage the file to current commit (HEAD): - ```shell - git reset HEAD <file> - ``` + ```shell + git reset HEAD <file> + ``` - Unstage everything - retain changes: - ```shell - git reset - ``` + ```shell + git reset + ``` - Discard all local changes, but save them for [later](#quickly-save-local-changes): - ```shell - git stash - ``` + ```shell + git stash + ``` - Discard everything permanently: - ```shell - git reset --hard - ``` + ```shell + git reset --hard + ``` ## Committed local changes @@ -240,21 +240,21 @@ In our example we will end up with commit `B`, that introduced bug/error. We hav - Undo (swap additions and deletions) changes introduced by commit `B`: - ```shell - git revert commit-B-id - ``` + ```shell + git revert commit-B-id + ``` - Undo changes on a single file or directory from commit `B`, but retain them in the staged state: - ```shell - git checkout commit-B-id <file> - ``` + ```shell + git checkout commit-B-id <file> + ``` - Undo changes on a single file or directory from commit `B`, but retain them in the unstaged state: - ```shell - git reset commit-B-id <file> - ``` + ```shell + git reset commit-B-id <file> + ``` - There is one command we also must not forget: **creating a new branch** from the point where changes are not applicable or where the development has hit a @@ -270,14 +270,14 @@ In our example we will end up with commit `B`, that introduced bug/error. We hav you can [cherry-pick](../../../user/project/merge_requests/cherry_pick_changes.md#cherry-picking-a-commit) that commit into a new merge request. -  +  - ```shell - git checkout commit-B-id - git checkout -b new-path-of-feature - # Create <commit F> - git commit -a - ``` + ```shell + git checkout commit-B-id + git checkout -b new-path-of-feature + # Create <commit F> + git commit -a + ``` ### With history modification @@ -297,9 +297,9 @@ delete commit `B`. - Rebase the range from current commit D to A: - ```shell - git rebase -i A - ``` + ```shell + git rebase -i A + ``` - Command opens your favorite editor where you write `drop` in front of commit `B`, but you leave default `pick` with all other commits. Save and exit the @@ -310,9 +310,9 @@ In case you want to modify something introduced in commit `B`. - Rebase the range from current commit D to A: - ```shell - git rebase -i A - ``` + ```shell + git rebase -i A + ``` - Command opens your favorite text editor where you write `edit` in front of commit `B`, but leave default `pick` with all other commits. Save and exit the editor to @@ -320,9 +320,9 @@ In case you want to modify something introduced in commit `B`. - Now do your edits and commit changes: - ```shell - git commit -a - ``` + ```shell + git commit -a + ``` You can find some more examples in [below section where we explain how to modify history](#how-modifying-history-is-done) diff --git a/doc/topics/git/partial_clone.md b/doc/topics/git/partial_clone.md new file mode 100644 index 00000000000..ea4223355d8 --- /dev/null +++ b/doc/topics/git/partial_clone.md @@ -0,0 +1,147 @@ +# Partial Clone for Large Repositories + +CAUTION: **Alpha:** +Partial Clone is an experimental feature, and will significantly increase +Gitaly resource utilization when performing a partial clone, and decrease +performance of subsequent fetch operations. + +As Git repositories become very large, usability decreases as performance +decreases. One major challenge is cloning the repository, because Git will +download the entire repository including every commit and every version of +every object. This can be slow to transfer, and require large amounts of disk +space. + +Historically, performing a **shallow clone** +([`--depth`](https://www.git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt)) +has been the only way to reduce the amount of data transferred when cloning +a Git repository. This does not, however, allow filtering by sub-tree which is +important for monolithic repositories containing many projects, or by object +size preventing unnecessary large objects being downloaded. + +[Partial clone](https://github.com/git/git/blob/master/Documentation/technical/partial-clone.txt) +is a performance optimization that "allows Git to function without having a +complete copy of the repository. The goal of this work is to allow Git better +handle extremely large repositories." + +Specifically, using partial clone, it should be possible for Git to natively +support: + +- large objects, instead of using [Git LFS](https://git-lfs.github.com/) +- enormous repositories + +Briefly, partial clone works by: + +- excluding objects from being transferred when cloning or fetching a + repository using a new `--filter` flag +- downloading missing objects on demand + +Follow [Git for enormous repositories](https://gitlab.com/groups/gitlab-org/-/epics/773) for roadmap and updates. + +## Enabling partial clone + +GitLab 12.1 uses Git 2.21.0 which has an arbitrary file access security +vulnerability when `uploadpack.allowFilter` is enabled, and should not be +enabled in production environments. + +A feature flag is planned to enable `uploadpack.allowFilter` and +`uploadpack.allowAnySHA1InWant` once the version of Git used by GitLab has been +updated to Git 2.22.0. + +Follow [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1553) for +updated. + +## Excluding objects by size + +Partial Clone allows large objects to be stored directly in the Git repository, +and be excluded from clones as desired by the user. This eliminates the error +prone process of deciding which objects should be stored in LFS or not. Using +partial clone, all files – large or small – may be treated the same. + +With the `uploadpack.allowFilter` and `uploadpack.allowAnySHA1InWant` options +enabled on the Git server: + +```bash +# clone the repo, excluding blobs larger than 1 megabyte +git clone --filter=blob:limit=1m <url> + +# in the checkout step of the clone, and any subsequent operations +# any blobs that are needed will be downloaded on demand +git checkout feature-branch +``` + +## Excluding objects by path + +Partial Clone allows clones to be filtered by path using a format similar to a +`.gitignore` file stored inside the repository. + +With the `uploadpack.allowFilter` and `uploadpack.allowAnySHA1InWant` options +enabled on the Git server: + +1. **Create a filter spec.** For example, consider a monolithic repository with + many applications, each in a different subdirectory in the root. Create a file + `shiny-app/.filterspec` using the GitLab web interface: + + ```.gitignore + # Only the paths listed in the file will be downloaded when performing a + # partial clone using `--filter=sparse:oid=shiny-app/.gitfilterspec` + + # Explicitly include filterspec needed to configure sparse checkout with + # git config --local core.sparsecheckout true + # git show master:snazzy-app/.gitfilterspec >> .git/info/sparse-checkout + shiny-app/.gitfilterspec + + # Shiny App + shiny-app/ + + # Dependencies + shimmery-app/ + shared-component-a/ + shared-component-b/ + ``` + +1. *Create a new Git repository and fetch.* Support for `--filter=sparse:oid` + using the clone command is incomplete, so we will emulate the clone command + by hand, using `git init` and `git fetch`. Follow + [gitaly#1769](https://gitlab.com/gitlab-org/gitaly/issues/1769) for updates. + + ```bash + # Create a new directory for the Git repository + mkdir jumbo-repo && cd jumbo-repo + + # Initialize a new Git repository + git init + + # Add the remote + git remote add origin git@gitlab.com/example/jumbo-repo + + # Enable partial clone support for the remote + git config --local extensions.partialClone origin + + # Fetch the filtered set of objects using the filterspec stored on the + # server. WARNING: this step is slow! + git fetch --filter=sparse:oid=master:shiny-app/.gitfilterspec origin + + # Optional: observe there are missing objects that we have not fetched + git rev-list --all --quiet --objects --missing=print | wc -l + ``` + + CAUTION: **IDE and Shell integrations:** + Git integrations with `bash`, `zsh`, etc and editors that automatically + show Git status information often run `git fetch` which will fetch the + entire repository. You many need to disable or reconfigure these + integrations. + +1. **Sparse checkout** must be enabled and configured to prevent objects from + other paths being downloaded automatically when checking out branches. Follow + [gitaly#1765](https://gitlab.com/gitlab-org/gitaly/issues/1765) for updates. + + ```bash + # Enable sparse checkout + git config --local core.sparsecheckout true + + # Configure sparse checkout + git show master:snazzy-app/.gitfilterspec >> .git/info/sparse-checkout + + # Checkout master + git checkout master + ``` diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index 417d91bf834..11284da30af 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -51,11 +51,11 @@ Configuring *both* the client and the server is unnecessary. - On UNIX, edit `~/.ssh/config` (create the file if it doesn’t exist) and add or edit: - ```text - Host your-gitlab-instance-url.com - ServerAliveInterval 60 - ServerAliveCountMax 5 - ``` + ```text + Host your-gitlab-instance-url.com + ServerAliveInterval 60 + ServerAliveCountMax 5 + ``` - On Windows, if you are using PuTTY, go to your session properties, then navigate to "Connection" and under "Sending of null packets to keep diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md new file mode 100644 index 00000000000..030e62f485a --- /dev/null +++ b/doc/topics/git/useful_git_commands.md @@ -0,0 +1,210 @@ +--- +type: reference +--- + +# Useful Git commands + +Here are some useful Git commands collected by the GitLab support team. You may not +need to use often, but they can can come in handy when needed. + +## Remotes + +### Add another URL to a remote, so both remotes get updated on each push + +```sh +git remote set-url --add <remote_name> <remote_url> +``` + +## Staging and reverting changes + +### Remove last commit and leave the changes in unstaged + +```sh +git reset --soft HEAD^ +``` + +### Unstage a certain number of commits from HEAD + +To unstage 3 commits, for example, run: + +```sh +git reset HEAD^3 +``` + +### Unstage changes to a certain file from HEAD + +```sh +git reset <filename> +``` + +### Revert a file to HEAD state and remove changes + +There are two options to revert changes to a file: + +- `git checkout <filename>` +- `git reset --hard <filename>` + +### Undo a previous commit by creating a new replacement commit + +```sh +git revert <commit-sha> +``` + +### Create a new message for last commit + +```sh +git commit --amend +``` + +### Add a file to the last commit + +```sh +git add <filename> +git commit --amend +``` + +Append `--no-edit` to the `commit` command if you do not want to edit the commit +message. + +## Stashing + +### Stash changes + +```sh +git stash save +``` + +The default behavor of `stash` is to save, so you can also use just: + +```sh +git stash +``` + +### Unstash your changes + +```sh +git stash apply +``` + +### Discard your stashed changes + +```sh +git stash drop +``` + +### Apply and drop your stashed changes + +```sh +git stash pop +``` + +## Refs and Log + +### Use reflog to show the log of reference changes to HEAD + +```sh +git reflog +``` + +### Check the Git history of a file + +The basic command to check the git history of a file: + +```sh +git log <file> +``` + +If you get this error message: + +```text +fatal: ambiguous argument <file_name>: unknown revision or path not in the working tree. +Use '--' to separate paths from revisions, like this: +``` + +Use this to check the Git history of the file: + +```sh +git log -- <file> +``` + +### Find the tags that contain a particular SHA + +```sh +git tag --contains <sha> +``` + +### Check the content of each change to a file + +```sh +gitk <file> +``` + +### Check the content of each change to a file, follows it past file renames + +```sh +gitk --follow <file> +``` + +## Debugging + +### Use a custom SSH key for a git command + +```sh +GIT_SSH_COMMAND="ssh -i ~/.ssh/gitlabadmin" git <command> +``` + +### Debug cloning + +With SSH: + +```sh +GIT_SSH_COMMAND="ssh -vvv" git clone <git@url> +``` + +With HTTPS: + +```sh +GIT_TRACE_PACKET=1 GIT_TRACE=2 GIT_CURL_VERBOSE=1 git clone <url> +``` + +## Rebasing + +### Rebase your branch onto master + +The -i flag stands for 'interactive': + +```sh +git rebase -i master +``` + +### Continue the rebase if paused + +```sh +git rebase --continue +``` + +### Use git rerere + +To _reuse_ recorded solutions to the same problems when repeated: + +```sh +git rerere +``` + +To enable `rerere` functionality: + +```sh +git config --global rerere.enabled true +``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/university/README.md b/doc/university/README.md index f696db2df20..25f77906e68 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -9,6 +9,10 @@ GitLab University is a great place to start when learning about version control If you're looking for a GitLab subscription for _your university_, see our [Education](https://about.gitlab.com/solutions/education/) page. +CAUTION: **Caution:** +Some of the content in GitLab University may be out of date and we plan to +[evaluate](https://gitlab.com/gitlab-org/gitlab-ce/issues/41064) it. + The GitLab University curriculum is composed of GitLab videos, screencasts, presentations, projects and external GitLab content hosted on other services and has been organized into the following sections: 1. [GitLab Beginner](#1-gitlab-beginner). @@ -53,7 +57,7 @@ The GitLab University curriculum is composed of GitLab videos, screencasts, pres 1. [Migrating from SVN](../user/project/import/svn.md) 1. [Migrating from Fogbugz](../user/project/import/fogbugz.md) -### 1.6. GitLab Inc. +### 1.6. The GitLab team 1. [About GitLab](https://about.gitlab.com/about/) 1. [GitLab Direction](https://about.gitlab.com/direction/) diff --git a/doc/university/training/end-user/README.md b/doc/university/training/end-user/README.md index 99fb5e83387..1218465c87a 100644 --- a/doc/university/training/end-user/README.md +++ b/doc/university/training/end-user/README.md @@ -9,12 +9,8 @@ which can be found at [End User Slides](https://gitlab-org.gitlab.io/end-user-tr through it's [RevealJS](https://gitlab.com/gitlab-org/end-user-training-slides) project. ---- - ## Git Intro ---- - ### What is a Version Control System (VCS) - Records changes to a file @@ -22,8 +18,6 @@ project. - Disaster Recovery - Types of VCS: Local, Centralized and Distributed ---- - ### Short Story of Git - 1991-2002: The Linux kernel was being maintained by sharing archived files @@ -31,8 +25,6 @@ project. - 2002: The Linux kernel project began using a DVCS called BitKeeper - 2005: BitKeeper revoked the free-of-charge status and Git was created ---- - ### What is Git - Distributed Version Control System @@ -42,8 +34,6 @@ project. - Disaster recovery friendly - Open Source ---- - ### Getting Help - Use the tools at your disposal when you get stuck. @@ -51,12 +41,9 @@ project. - Use Google (i.e. StackOverflow, Google groups) - Read documentation at <https://git-scm.com> ---- - ## Git Setup -Workshop Time! ---- +Workshop Time! ### Setup @@ -68,8 +55,6 @@ Workshop Time! - Debian: `sudo apt-get install git-all` - Red Hat `sudo yum install git-all` ---- - ### Configure - One-time configuration of the Git client: @@ -88,9 +73,7 @@ git config --global --list ``` - You might want or be required to use an SSH key. - - Instructions: [SSH](http://doc.gitlab.com/ce/ssh/README.html) - ---- + - Instructions: [SSH](http://doc.gitlab.com/ce/ssh/README.html) ### Workspace @@ -98,8 +81,6 @@ git config --global --list - Create a workspace or development directory - This is where we'll be working and adding content ---- - ```bash mkdir ~/development cd ~/development @@ -110,24 +91,18 @@ mkdir ~/workspace cd ~/workspace ``` ---- - ## Git Basics ---- - ### Git Workflow - Untracked files - - New files that Git has not been told to track previously. + - New files that Git has not been told to track previously. - Working area (Workspace) - - Files that have been modified but are not committed. + - Files that have been modified but are not committed. - Staging area (Index) - - Modified files that have been marked to go in the next commit. + - Modified files that have been marked to go in the next commit. - Upstream - - Hosted repository on a shared server - ---- + - Hosted repository on a shared server ### GitLab @@ -136,8 +111,6 @@ cd ~/workspace issue tracking, Merge Requests, and other features. - The hosted version of GitLab is gitlab.com ---- - ### New Project - Sign in into your gitlab.com account @@ -145,8 +118,6 @@ cd ~/workspace - Choose to import from 'Any Repo by URL' and use <https://gitlab.com/gitlab-org/training-examples.git> - On your machine clone the `training-examples` project ---- - ### Git and GitLab basics 1. Edit `edit_this_file.rb` in `training-examples` @@ -157,8 +128,6 @@ cd ~/workspace 1. Push the commit to the remote 1. View the git log ---- - ```shell # Edit `edit_this_file.rb` git status @@ -169,8 +138,6 @@ git push origin master git log ``` ---- - ### Feature Branching 1. Create a new feature branch called `squash_some_bugs` @@ -178,8 +145,6 @@ git log 1. Commit 1. Push ---- - ```shell git checkout -b squash_some_bugs # Edit `bugs.rb` @@ -189,14 +154,8 @@ git commit -m 'Fix some buggy code' git push origin squash_some_bugs ``` ---- - ## Merge Request ---- - -### Merge requests - - When you want feedback create a merge request - Target is the ‘default’ branch (usually master) - Assign or mention the person you would like to review @@ -205,8 +164,6 @@ git push origin squash_some_bugs - Anyone can comment, not just the assignee - Push corrections to the same branch ---- - ### Merge request example - Create your first merge request @@ -215,8 +172,6 @@ git push origin squash_some_bugs - Push a new commit to the same branch - Review the changes again and notice the update ---- - ### Feedback and Collaboration - Merge requests are a time for feedback and collaboration @@ -229,26 +184,17 @@ git push origin squash_some_bugs --- -### Feedback and Collaboration - -- Review the Thoughtbot code-review guide for suggestions to follow when reviewing merge requests:[Thoughtbot](https://github.com/thoughtbot/guides/tree/master/code-review) +- Review the Thoughtbot code-review guide for suggestions to follow when reviewing merge requests: + [Thoughtbot](https://github.com/thoughtbot/guides/tree/master/code-review) - See GitLab merge requests for examples: [Merge Requests](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests) ---- - ## Merge Conflicts ---- - -### Merge Conflicts - - Happen often - Learning to fix conflicts is hard - Practice makes perfect - Force push after fixing conflicts. Be careful! ---- - ### Example Plan 1. Checkout a new branch and edit conflicts.rb. Add 'Line4' and 'Line5'. @@ -262,47 +208,45 @@ git push origin squash_some_bugs 1. Force push the changes 1. Finally continue with the Merge Request ---- - ### Example 1/2 - git checkout -b conflicts_branch +```sh +git checkout -b conflicts_branch - # vi conflicts.rb - # Add 'Line4' and 'Line5' +# vi conflicts.rb +# Add 'Line4' and 'Line5' - git commit -am "add line4 and line5" - git push origin conflicts_branch +git commit -am "add line4 and line5" +git push origin conflicts_branch - git checkout master +git checkout master - # vi conflicts.rb - # Add 'Line6' and 'Line7' - git commit -am "add line6 and line7" - git push origin master - ---- +# vi conflicts.rb +# Add 'Line6' and 'Line7' +git commit -am "add line6 and line7" +git push origin master +``` ### Example 2/2 Create a merge request on the GitLab web UI. You'll see a conflict warning. - git checkout conflicts_branch - git fetch - git rebase master - - # Fix conflicts by editing the files. +```sh +git checkout conflicts_branch +git fetch +git rebase master - git add conflicts.rb - # No need to commit this file +# Fix conflicts by editing the files. - git rebase --continue +git add conflicts.rb +# No need to commit this file - # Remember that we have rewritten our commit history so we - # need to force push so that our remote branch is restructured - git push origin conflicts_branch -f +git rebase --continue ---- +# Remember that we have rewritten our commit history so we +# need to force push so that our remote branch is restructured +git push origin conflicts_branch -f +``` ### Notes @@ -311,55 +255,63 @@ Create a merge request on the GitLab web UI. You'll see a conflict warning. - Merge when bringing changes from feature to master - Reference: <https://www.atlassian.com/git/tutorials/merging-vs-rebasing/> ---- - ## Revert and Unstage ---- - ### Unstage To remove files from stage use reset HEAD. Where HEAD is the last commit of the current branch: - git reset HEAD <file> +```sh +git reset HEAD <file> +``` This will unstage the file but maintain the modifications. To revert the file back to the state it was in before the changes we can use: - git checkout -- <file> +```sh +git checkout -- <file> +``` To remove a file from disk and repo use 'git rm' and to rm a dir use the '-r' flag: - git rm '*.txt' - git rm -r <dirname> +```sh +git rm '*.txt' +git rm -r <dirname> +``` If we want to remove a file from the repository but keep it on disk, say we forgot to add it to our .gitignore file then use `--cache`: - git rm <filename> --cache - ---- +```sh +git rm <filename> --cache +``` ### Undo Commits Undo last commit putting everything back into the staging area: - git reset --soft HEAD^ +```sh +git reset --soft HEAD^ +``` Add files and change message with: - git commit --amend -m "New Message" +```sh +git commit --amend -m "New Message" +``` Undo last and remove changes - git reset --hard HEAD^ +```sh +git reset --hard HEAD^ +``` Same as last one but for two commits back: - git reset --hard HEAD^^ +```sh +git reset --hard HEAD^^ +``` Don't reset after pushing ---- - ### Reset Workflow 1. Edit file again 'edit_this_file.rb' @@ -373,53 +325,46 @@ Don't reset after pushing 1. Pull for updates 1. Push changes ----- - - # Change file edit_this_file.rb - git status - git commit -am "kjkfjkg" - git log - git commit --amend -m "New comment added" - git log - git reset --soft HEAD^ - git log - git pull origin master - git push origin master - ---- +```sh +# Change file edit_this_file.rb +git status +git commit -am "kjkfjkg" +git log +git commit --amend -m "New comment added" +git log +git reset --soft HEAD^ +git log +git pull origin master +git push origin master +``` -### Note +### git revert vs git reset -git revert vs git reset Reset removes the commit while revert removes the changes but leaves the commit Revert is safer considering we can revert a revert - # Changed file - git commit -am "bug introduced" - git revert HEAD - # New commit created reverting changes - # Now we want to re apply the reverted commit - git log # take hash from the revert commit - git revert <rev commit hash> - # reverted commit is back (new commit created again) - ---- +```sh +# Changed file +git commit -am "bug introduced" +git revert HEAD +# New commit created reverting changes +# Now we want to re apply the reverted commit +git log # take hash from the revert commit +git revert <rev commit hash> +# reverted commit is back (new commit created again) +``` ## Questions ---- - ## Instructor Notes ---- - ### Version Control - - Local VCS was used with a filesystem or a simple db. - - Centralized VCS such as Subversion includes collaboration but - still is prone to data loss as the main server is the single point of - failure. - - Distributed VCS enables the team to have a complete copy of the project - and work with little dependency to the main server. In case of a main - server failing the project can be recovered by any of the latest copies - from the team +- Local VCS was used with a filesystem or a simple db. +- Centralized VCS such as Subversion includes collaboration but + still is prone to data loss as the main server is the single point of + failure. +- Distributed VCS enables the team to have a complete copy of the project + and work with little dependency to the main server. In case of a main + server failing the project can be recovered by any of the latest copies + from the team diff --git a/doc/university/training/topics/bisect.md b/doc/university/training/topics/bisect.md index 4178afa2086..24dc670d9d5 100644 --- a/doc/university/training/topics/bisect.md +++ b/doc/university/training/topics/bisect.md @@ -4,13 +4,11 @@ comments: false # Bisect -## Bisect - - Find a commit that introduced a bug - Works through a process of elimination - Specify a known good and bad revision to begin -## Bisect +## Bisect sample workflow 1. Start the bisect process 1. Enter the bad revision (usually latest commit) diff --git a/doc/university/training/topics/cherry_picking.md b/doc/university/training/topics/cherry_picking.md index fa0cb5fe6a4..f5bcdfcbf12 100644 --- a/doc/university/training/topics/cherry_picking.md +++ b/doc/university/training/topics/cherry_picking.md @@ -4,13 +4,11 @@ comments: false # Cherry Pick -## Cherry Pick - - Given an existing commit on one branch, apply the change to another branch - Useful for backporting bug fixes to previous release branches - Make the commit on the master branch and pick in to stable -## Cherry Pick +## Cherry Pick sample workflow 1. Check out a new 'stable' branch from 'master' 1. Change back to 'master' @@ -19,8 +17,6 @@ comments: false 1. Check out the 'stable' branch 1. Cherry pick the commit using the SHA obtained earlier -## Commands - ```bash git checkout master git checkout -b stable diff --git a/doc/university/training/topics/env_setup.md b/doc/university/training/topics/env_setup.md index 305f5ecb1fb..92d2613c5d2 100644 --- a/doc/university/training/topics/env_setup.md +++ b/doc/university/training/topics/env_setup.md @@ -14,9 +14,11 @@ comments: false - If it's not installed, it will prompt you to install it. - **Linux** + ```bash sudo yum install git-all ``` + ```bash sudo apt-get install git-all ``` diff --git a/doc/university/training/topics/feature_branching.md b/doc/university/training/topics/feature_branching.md index d2efe634533..f530389d4da 100644 --- a/doc/university/training/topics/feature_branching.md +++ b/doc/university/training/topics/feature_branching.md @@ -11,15 +11,13 @@ comments: false - Push branches to the server frequently - Hint: This is a cheap backup for your work-in-progress code -## Feature branching +## Feature branching sample workflow 1. Create a new feature branch called 'squash_some_bugs' 1. Edit '`bugs.rb`' and remove all the bugs. 1. Commit 1. Push -## Commands - ```sh git checkout -b squash_some_bugs # Edit `bugs.rb` diff --git a/doc/university/training/topics/getting_started.md b/doc/university/training/topics/getting_started.md index 08027c5d15b..3fadb58e804 100644 --- a/doc/university/training/topics/getting_started.md +++ b/doc/university/training/topics/getting_started.md @@ -8,14 +8,15 @@ comments: false - Create a new repository by instantiating it through: - ```bash - git init - ``` + ```bash + git init + ``` + - Copy an existing project by cloning the repository through: - ```bash - git clone <url> - ``` + ```bash + git clone <url> + ``` ## Central Repos @@ -23,9 +24,9 @@ comments: false - Bare repositories don't allow file editing or committing changes. - Create a bare repo with: - ```bash - git init --bare project-name.git - ``` + ```bash + git init --bare project-name.git + ``` ## Instantiate workflow with clone @@ -34,8 +35,6 @@ comments: false 1. Create a '`Workspace`' directory in your home directory. 1. Clone the '`training-examples`' project. -## Commands - ```sh mkdir ~/workspace cd ~/workspace @@ -68,8 +67,6 @@ Modified files that have been marked to go in the next commit. 1. Push the commit to the remote 1. View the git log -## Commands - ```sh # Edit `edit_this_file.rb` git status diff --git a/doc/university/training/topics/git_add.md b/doc/university/training/topics/git_add.md index 4c61d5eb175..0c9a50bb5e1 100644 --- a/doc/university/training/topics/git_add.md +++ b/doc/university/training/topics/git_add.md @@ -4,38 +4,34 @@ comments: false # Git Add -## Git Add - Adds content to the index or staging area. - Adds a list of file: - ```bash - git add <files> - ``` + ```bash + git add <files> + ``` - Adds all files including deleted ones: - ```bash - git add -A - ``` - -## Git add continued + ```bash + git add -A + ``` - Add all text files in current dir: - ```bash - git add *.txt - ``` + ```bash + git add *.txt + ``` - Add all text file in the project: - ```bash - git add "*.txt*" - ``` + ```bash + git add "*.txt*" + ``` - Adds all files in directory: - ```bash - git add views/layouts/ - ``` + ```bash + git add views/layouts/ + ``` diff --git a/doc/university/training/topics/git_log.md b/doc/university/training/topics/git_log.md index 11addcd3ee1..bae734554f5 100644 --- a/doc/university/training/topics/git_log.md +++ b/doc/university/training/topics/git_log.md @@ -8,33 +8,33 @@ Git log lists commit history. It allows searching and filtering. - Initiate log: - ```sh - git log - ``` + ```sh + git log + ``` - Retrieve set number of records: - ```sh - git log -n 2 - ``` + ```sh + git log -n 2 + ``` - Search commits by author. Allows user name or a regular expression. - ```sh - git log --author="user_name" - ``` + ```sh + git log --author="user_name" + ``` - Search by comment message: - ```sh - git log --grep="<pattern>" - ``` + ```sh + git log --grep="<pattern>" + ``` - Search by date: - ```sh - git log --since=1.month.ago --until=3.weeks.ago - ``` + ```sh + git log --since=1.month.ago --until=3.weeks.ago + ``` ## Git Log Workflow diff --git a/doc/university/training/topics/merge_conflicts.md b/doc/university/training/topics/merge_conflicts.md index dd235fe3a81..97bb038f405 100644 --- a/doc/university/training/topics/merge_conflicts.md +++ b/doc/university/training/topics/merge_conflicts.md @@ -9,7 +9,7 @@ comments: false - Practice makes perfect - Force push after fixing conflicts. Be careful! -## Merge conflicts +## Merge conflicts sample workflow 1. Checkout a new branch and edit `conflicts.rb`. Add 'Line4' and 'Line5'. 1. Commit and push. @@ -22,8 +22,6 @@ comments: false 1. Force push the changes. 1. Finally continue with the Merge Request. -## Commands - ```sh git checkout -b conflicts_branch diff --git a/doc/university/training/topics/merge_requests.md b/doc/university/training/topics/merge_requests.md index b5bbe7b2e1e..656871ae5b2 100644 --- a/doc/university/training/topics/merge_requests.md +++ b/doc/university/training/topics/merge_requests.md @@ -30,8 +30,6 @@ comments: false - Be as receptive as possible - Feedback is about the best code, not the person. You are not your code -## Feedback and Collaboration - Review the Thoughtbot code-review guide for suggestions to follow when reviewing merge requests: [https://github.com/thoughtbot/guides/tree/master/code-review](https://github.com/thoughtbot/guides/tree/master/code-review) diff --git a/doc/university/training/topics/rollback_commits.md b/doc/university/training/topics/rollback_commits.md index 1e6602deef2..c17e8d59737 100644 --- a/doc/university/training/topics/rollback_commits.md +++ b/doc/university/training/topics/rollback_commits.md @@ -8,29 +8,29 @@ comments: false - Undo last commit putting everything back into the staging area: - ```sh - git reset --soft HEAD^ - ``` + ```sh + git reset --soft HEAD^ + ``` - Add files and change message with: - ```sh - git commit --amend -m "New Message" - ``` + ```sh + git commit --amend -m "New Message" + ``` - Undo last and remove changes: - ```sh - git reset --hard HEAD^ - ``` + ```sh + git reset --hard HEAD^ + ``` - Same as last one but for two commits back: - ```sh - git reset --hard HEAD^^ - ``` + ```sh + git reset --hard HEAD^^ + ``` -** Don't reset after pushing ** +**Don't reset after pushing** ## Reset Workflow diff --git a/doc/university/training/topics/stash.md b/doc/university/training/topics/stash.md index 02b2f610266..d3e63db0c6a 100644 --- a/doc/university/training/topics/stash.md +++ b/doc/university/training/topics/stash.md @@ -9,52 +9,52 @@ and we need to change to a different branch. - Stash: - ```sh - git stash save - # or - git stash - # or with a message - git stash save "this is a message to display on the list" - ``` + ```sh + git stash save + # or + git stash + # or with a message + git stash save "this is a message to display on the list" + ``` - Apply stash to keep working on it: - ```sh - git stash apply - # or apply a specific one from out stack - git stash apply stash@{3} - ``` + ```sh + git stash apply + # or apply a specific one from out stack + git stash apply stash@{3} + ``` -- Every time we save a stash it gets stacked so by using list we can see all our +- Every time we save a stash it gets stacked so by using `list` we can see all our stashes. - ```sh - git stash list - # or for more information (log methods) - git stash list --stat - ``` + ```sh + git stash list + # or for more information (log methods) + git stash list --stat + ``` - To clean our stack we need to manually remove them: - ```sh - # drop top stash - git stash drop - # or - git stash drop <name> - # to clear all history we can use - git stash clear - ``` + ```sh + # drop top stash + git stash drop + # or + git stash drop <name> + # to clear all history we can use + git stash clear + ``` - Apply and drop on one command: - ```sh - git stash pop - ``` + ```sh + git stash pop + ``` - If we meet conflicts we need to either reset or commit our changes. - Conflicts through `pop` will not drop a stash afterwards. -## Git Stash +## Git Stash sample workflow 1. Modify a file 1. Stage file @@ -64,8 +64,6 @@ and we need to change to a different branch. 1. Apply with pop 1. View list to confirm changes -## Commands - ```sh # Modify edit_this_file.rb file git add . diff --git a/doc/university/training/topics/tags.md b/doc/university/training/topics/tags.md index cdbb8a2da7c..631b93cc384 100644 --- a/doc/university/training/topics/tags.md +++ b/doc/university/training/topics/tags.md @@ -11,18 +11,12 @@ type: reference - Many projects combine an annotated release tag with a stable branch - Consider setting deployment/release tags automatically -# Tags +## Tags sample workflow - Create a lightweight tag - Create an annotated tag - Push the tags to the remote repository -**Additional resources** - -<https://git-scm.com/book/en/Git-Basics-Tagging> - -# Commands - ```sh git checkout master @@ -36,6 +30,10 @@ git tag git push origin --tags ``` +**Additional resources** + +<https://git-scm.com/book/en/Git-Basics-Tagging> + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/university/training/topics/unstage.md b/doc/university/training/topics/unstage.md index af16afdc5d1..d7482bf2bd5 100644 --- a/doc/university/training/topics/unstage.md +++ b/doc/university/training/topics/unstage.md @@ -4,29 +4,27 @@ comments: false # Unstage -## Unstage +- To remove files from stage use reset HEAD where HEAD is the last commit of the current branch. This will unstage the file but maintain the modifications. -- To remove files from stage use reset HEAD where HEAD is the last commit of the current branch. This will unstage the file but maintain the modifications. - - ```bash - git reset HEAD <file> - ``` + ```bash + git reset HEAD <file> + ``` - To revert the file back to the state it was in before the changes we can use: - ```bash - git checkout -- <file> - ``` + ```bash + git checkout -- <file> + ``` - To remove a file from disk and repo use 'git rm' and to rm a dir use the '-r' flag: - ```sh - git rm '*.txt' - git rm -r <dirname> - ``` + ```sh + git rm '*.txt' + git rm -r <dirname> + ``` - If we want to remove a file from the repository but keep it on disk, say we forgot to add it to our `.gitignore` file then use `--cache`: - ```sh - git rm <filename> --cache - ``` + ```sh + git rm <filename> --cache + ``` diff --git a/doc/update/README.md b/doc/update/README.md index 974982da5d0..42c43110a19 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -135,6 +135,30 @@ If you need to downgrade your Enterprise Edition installation back to Community Edition, you can follow [this guide][ee-ce] to make the process as smooth as possible. +## Version specific upgrading instructions + +### 12.2.0 + +In 12.2.0, we enabled Rails' authenticated cookie encryption. Old sessions are +automatically upgraded. + +However, session cookie downgrades are not supported. So after upgrading to 12.2.0, +any downgrades would result to all sessions being invalidated and users are logged out. + +### 12.0.0 + +In 12.0.0 we made various database related changes. These changes require that +users first upgrade to the latest 11.11 patch release. Once upgraded to 11.11.x, +users can upgrade to 12.x. Failure to do so may result in database migrations +not being applied, which could lead to application errors. + +Example 1: you are currently using GitLab 11.11.3, which is the latest patch +release for 11.11.x. You can upgrade as usual to 12.0.0, 12.1.0, etc. + +Example 2: you are currently using a version of GitLab 10.x. To upgrade, first +upgrade to 11.11.3. Once upgraded to 11.11.3 you can safely upgrade to 12.0.0 +or future versions. + ## Miscellaneous - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index 1e424134242..f6a1b6abdbf 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -72,15 +72,15 @@ need to enable the bundled PostgreSQL: 1. Stop GitLab: - ```bash - sudo gitlab-ctl stop - ``` + ```bash + sudo gitlab-ctl stop + ``` 1. Edit `/etc/gitlab/gitlab.rb` to enable bundled PostgreSQL: - ``` - postgresql['enable'] = true - ``` + ``` + postgresql['enable'] = true + ``` 1. Edit `/etc/gitlab/gitlab.rb` to use the bundled PostgreSQL. Please check all the settings beginning with `db_`, such as `gitlab_rails['db_adapter']` @@ -91,22 +91,22 @@ need to enable the bundled PostgreSQL: for the changes to take effect. 1. Start Unicorn and PostgreSQL so that we can prepare the schema: - ```bash - sudo gitlab-ctl start unicorn - sudo gitlab-ctl start postgresql - ``` + ```bash + sudo gitlab-ctl start unicorn + sudo gitlab-ctl start postgresql + ``` 1. Run the following commands to prepare the schema: - ```bash - sudo gitlab-rake db:create db:migrate - ``` + ```bash + sudo gitlab-rake db:create db:migrate + ``` 1. Stop Unicorn to prevent other database access from interfering with the loading of data: - ```bash - sudo gitlab-ctl stop unicorn - ``` + ```bash + sudo gitlab-ctl stop unicorn + ``` After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. @@ -116,57 +116,57 @@ new PostgreSQL one: 1. Save the following snippet in a `commands.load` file, and edit with your MySQL database `username`, `password` and `host`: - ``` - LOAD DATABASE - FROM mysql://username:password@host/gitlabhq_production - INTO postgresql://gitlab-psql@unix://var/opt/gitlab/postgresql:/gitlabhq_production + ``` + LOAD DATABASE + FROM mysql://username:password@host/gitlabhq_production + INTO postgresql://gitlab-psql@unix://var/opt/gitlab/postgresql:/gitlabhq_production - WITH include no drop, truncate, disable triggers, create no tables, - create no indexes, preserve index names, no foreign keys, - data only + WITH include no drop, truncate, disable triggers, create no tables, + create no indexes, preserve index names, no foreign keys, + data only - ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' + ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' - ; - ``` + ; + ``` 1. Start the migration: - ```bash - sudo -u gitlab-psql pgloader commands.load - ``` + ```bash + sudo -u gitlab-psql pgloader commands.load + ``` 1. Once the migration finishes, you should see a summary table that looks like the following: - ``` - table name read imported errors total time - ----------------------------------------------- --------- --------- --------- -------------- - fetch meta data 119 119 0 0.388s - Truncate 119 119 0 1.134s - ----------------------------------------------- --------- --------- --------- -------------- - public.abuse_reports 0 0 0 0.490s - public.appearances 0 0 0 0.488s - . - . - . - public.web_hook_logs 0 0 0 1.080s - ----------------------------------------------- --------- --------- --------- -------------- - COPY Threads Completion 4 4 0 2.008s - Reset Sequences 113 113 0 0.304s - Install Comments 0 0 0 0.000s - ----------------------------------------------- --------- --------- --------- -------------- - Total import time 1894 1894 0 12.497s - ``` - - If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See - the [troubleshooting guide](#troubleshooting) for more details. + ``` + table name read imported errors total time + ----------------------------------------------- --------- --------- --------- -------------- + fetch meta data 119 119 0 0.388s + Truncate 119 119 0 1.134s + ----------------------------------------------- --------- --------- --------- -------------- + public.abuse_reports 0 0 0 0.490s + public.appearances 0 0 0 0.488s + . + . + . + public.web_hook_logs 0 0 0 1.080s + ----------------------------------------------- --------- --------- --------- -------------- + COPY Threads Completion 4 4 0 2.008s + Reset Sequences 113 113 0 0.304s + Install Comments 0 0 0 0.000s + ----------------------------------------------- --------- --------- --------- -------------- + Total import time 1894 1894 0 12.497s + ``` + + If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See + the [troubleshooting guide](#troubleshooting) for more details. 1. Start GitLab: - ```bash - sudo gitlab-ctl start - ``` + ```bash + sudo gitlab-ctl start + ``` You can now verify that everything works as expected by visiting GitLab. diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 0506d992d4b..6a41c6aec32 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -41,7 +41,7 @@ sudo -u git -H git checkout -- Gemfile.lock db/schema.rb locale sudo -u git -H git checkout LATEST_TAG -b LATEST_TAG ``` -### 3. Install libs, migrations, etc. +### 3. Install libs, migrations, etc ```bash cd /home/git/gitlab diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index bea5bcd9dd7..49e2fb2568e 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -54,7 +54,7 @@ sudo -u git -H git remote add -f ee https://gitlab.com/gitlab-org/gitlab-ee.git sudo -u git -H git checkout EE_BRANCH ``` -### 3. Install libs, migrations, etc. +### 3. Install libs, migrations, etc ```sh cd /home/git/gitlab diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index d3b0a3c2829..f6b64dcf694 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -47,8 +47,8 @@ sudo service gitlab stop ### 3. Update Ruby -NOTE: Beginning in GitLab 11.6, we only support Ruby 2.5 or higher, and dropped -support for Ruby 2.4. Be sure to upgrade if necessary. +NOTE: Beginning in GitLab 12.2, we only support Ruby 2.6 or higher, and dropped +support for Ruby 2.5. Be sure to upgrade if necessary. You can check which version you are running with `ruby -v`. @@ -313,7 +313,7 @@ For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload ``` -### 13. Install libs, migrations, etc. +### 13. Install libs, migrations, etc ```bash cd /home/git/gitlab @@ -378,20 +378,6 @@ Example: Additional instructions here. --> -### 12.0.0 - -In 12.0.0 we made various database related changes. These changes require that -users first upgrade to the latest 11.11 patch release. Once upgraded to 11.11.x, -users can upgrade to 12.x. Failure to do so may result in database migrations -not being applied, which could lead to application errors. - -Example 1: you are currently using GitLab 11.11.3, which is the latest patch -release for 11.11.x. You can upgrade as usual to 12.0.0, 12.1.0, etc. - -Example 2: you are currently using a version of GitLab 10.x. To upgrade, first -upgrade to 11.11.3. Once upgraded to 11.11.3 you can safely upgrade to 12.0.0 -or future versions. - ## Things went south? Revert to previous version ### 1. Revert the code to the previous version diff --git a/doc/user/abuse_reports.md b/doc/user/abuse_reports.md index 41ee7e62b2c..e6c86cc8f2e 100644 --- a/doc/user/abuse_reports.md +++ b/doc/user/abuse_reports.md @@ -1,6 +1,12 @@ # Abuse reports -Report abuse from users to GitLab administrators. +You can report abuse from other GitLab users to GitLab administrators. + +A GitLab administrator [can then choose](admin_area/abuse_reports.md) to: + +- Remove the user, which deletes them from the instance. +- Block the user, which denies them access to the instance. +- Or remove the report, which retains the users access to the instance. You can report a user through their: @@ -12,7 +18,8 @@ You can report a user through their: To report abuse from a user's profile page: -1. Click on the exclamation point report abuse button at the top right of the user's profile. +1. Click on the exclamation point report abuse button at the top right of the + user's profile. 1. Complete an abuse report. 1. Click the **Send report** button. @@ -26,15 +33,18 @@ To report abuse from a user's comment: 1. Click the **Send report** button. NOTE: **Note:** -A URL to the reported user's comment will be -pre-filled in the abuse report's **Message** field. +A URL to the reported user's comment will be pre-filled in the abuse report's +**Message** field. ## Reporting abuse through a user's issue or merge request The **Report abuse** button is displayed at the top right of the issue or merge request: -- When **Report abuse** is selected from the menu that appears when the **Close issue** or **Close merge request** button is clicked, for users that have permission to close the issue or merge request. -- When viewing the issue or merge request, for users that don't have permission to close the issue or merge request. +- When **Report abuse** is selected from the menu that appears when the + **Close issue** or **Close merge request** button is clicked, for users that + have permission to close the issue or merge request. +- When viewing the issue or merge request, for users that don't have permission + to close the issue or merge request. With the **Report abuse** button displayed, to submit an abuse report: diff --git a/doc/user/admin_area/abuse_reports.md b/doc/user/admin_area/abuse_reports.md index 01c2d9607f5..cd8dc7bcbf6 100644 --- a/doc/user/admin_area/abuse_reports.md +++ b/doc/user/admin_area/abuse_reports.md @@ -1,31 +1,77 @@ -# Abuse reports +--- +type: reference, howto +--- + +# Abuse reports **(CORE ONLY)** View and resolve abuse reports from GitLab users. -Admins can view abuse reports in the admin area and are able to -resolve the reports by removing the reported user, blocking the reported user, or removing the report. +GitLab administrators can view and [resolve](#resolving-abuse-reports) abuse +reports in the Admin Area. ## Reporting abuse -To find out more about reporting abuse, see [abuse reports user documentation](../abuse_reports.md). +To find out more about reporting abuse, see [abuse reports user +documentation](../abuse_reports.md). ## Resolving abuse reports -To access abuse reports, go to **Admin area > Abuse Reports**. +To access abuse reports, go to **Admin Area > Abuse Reports**. There are 3 ways to resolve an abuse report, with a button for each method: -- Remove user & report: [Deletes the reported user](../profile/account/delete_account.md) from the instance and removes the abuse report from the list. -- Block user: Blocks the reported user from the instance and does not remove the abuse report from the list. -- Remove report: Removes the abuse report from the list and does not restrict the access for the reported user. +- Remove user & report. This will: + - [Delete the reported user](../profile/account/delete_account.md) from the + instance. + - Remove the abuse report from the list. +- [Block user](#blocking-users). +- Remove report. This will: + - Remove the abuse report from the list. + - Remove access restrictions for the reported user. + +The following is an example of the **Abuse Reports** page:  -## Blocked users +### Blocking users + +A blocked user cannot log in or access any repositories, but all of their data +remains. + +Blocking a user: + +- Leaves them in the abuse report list. +- Changes the **Block user** button to a disabled **Already blocked** button. + +The user will be notified with the +[following message](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/email_receiver_worker.rb#L38): -Blocking a user will not remove the abuse report from the list. +```text +Your account has been blocked. If you believe this is in error, contact a staff member. +``` -Instead, the block button will be disabled and explain that the user is "Already blocked". -You are still able to remove the user and/or report if necessary. +After blocking, you can still either: + +- Remove the user and report if necessary. +- Remove the report. + +The following is an example of a blocked user listed on the **Abuse Reports** +page:  + +NOTE: **Note:** +Users can be [blocked](../../api/users.md#block-user) and +[unblocked](../../api/users.md#unblock-user) using the GitLab API. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/broadcast_messages.md b/doc/user/admin_area/broadcast_messages.md index 02445abdb37..b0491499f88 100644 --- a/doc/user/admin_area/broadcast_messages.md +++ b/doc/user/admin_area/broadcast_messages.md @@ -1,4 +1,8 @@ -# Broadcast Messages +--- +type: reference, howto +--- + +# Broadcast Messages **(CORE ONLY)** GitLab can display messages to all users of a GitLab instance in a banner that appears in the UI. @@ -51,3 +55,15 @@ Once deleted, the broadcast message is removed from the list of broadcast messag NOTE: **Note:** Broadcast messages can be deleted while active. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md index 427f3103cfc..02c2efaa4f3 100644 --- a/doc/user/admin_area/custom_project_templates.md +++ b/doc/user/admin_area/custom_project_templates.md @@ -1,26 +1,49 @@ +--- +type: reference +--- + # Custom instance-level project templates **(PREMIUM ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2. -When you create a new [project](../project/index.md), creating it based on custom project templates is -a convenient bootstrap option. +GitLab administrators can configure the group where all the custom project +templates are sourced. -GitLab administrators can configure a GitLab group that serves as template -source for an entire GitLab instance under **Admin area > Settings > Custom project templates**. +Every project directly under the group namespace will be +available to the user if they have access to them. For example: + +- Public project in the group will be available to every logged in user. +- Private projects will be available only if the user is a member of the project. + +Repository and database information that are copied over to each new project are +identical to the data exported with +[GitLab's Project Import/Export](../project/settings/import_export.md). NOTE: **Note:** To set project templates at a group level, see [Custom group-level project templates](../group/custom_project_templates.md). -Within this section, you can configure the group where all the custom project -templates are sourced. Every project directly under the group namespace will be -available to the user if they have access to them. For example, every public -project in the group will be available to every logged in user. +## Configuring -However, private projects will be available only if the user is a member of the project. +GitLab administrators can configure a GitLab group that serves as template +source for an entire GitLab instance by: + +1. Navigating to **Admin area > Settings > Templates**. +1. Expanding **Custom project templates**. +1. Selecting a group to use. +1. Pressing **Save changes**. NOTE: **Note:** Projects below subgroups of the template group are **not** supported. -Repository and database information that are copied over to each new project are -identical to the data exported with [GitLab's Project Import/Export](../project/settings/import_export.md). +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/diff_limits.md b/doc/user/admin_area/diff_limits.md index 4063c40a751..9fe4b50a991 100644 --- a/doc/user/admin_area/diff_limits.md +++ b/doc/user/admin_area/diff_limits.md @@ -2,7 +2,7 @@ type: reference --- -# Diff limits administration +# Diff limits administration **(CORE ONLY)** You can set a maximum size for display of diff files (patches). diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md index 9e7057f93d4..39753fd885e 100644 --- a/doc/user/admin_area/geo_nodes.md +++ b/doc/user/admin_area/geo_nodes.md @@ -64,8 +64,8 @@ Internal URL defaults to External URL, but you can customize it under CAUTION: **Warning:** We recommend using an HTTPS connection while configuring the Geo nodes. To avoid breaking communication between **primary** and **secondary** nodes when using -HTTPS, customize your Internal URL to point to a Load Balancer with TLS -termination. +HTTPS, customize your Internal URL to point to a load balancer with TLS +terminated at the load balancer. ## Multiple secondary nodes behind a load balancer diff --git a/doc/user/admin_area/img/abuse_report_blocked_user.png b/doc/user/admin_area/img/abuse_report_blocked_user.png Binary files differindex 0cb4c7bb8ac..435d8dfe821 100644 --- a/doc/user/admin_area/img/abuse_report_blocked_user.png +++ b/doc/user/admin_area/img/abuse_report_blocked_user.png diff --git a/doc/user/admin_area/img/abuse_reports_page.png b/doc/user/admin_area/img/abuse_reports_page.png Binary files differindex 81dbe976cda..30e932211cb 100644 --- a/doc/user/admin_area/img/abuse_reports_page.png +++ b/doc/user/admin_area/img/abuse_reports_page.png diff --git a/doc/user/admin_area/img/broadcast_messages.png b/doc/user/admin_area/img/broadcast_messages.png Binary files differindex 926d38ae049..f0ae92f8c17 100644 --- a/doc/user/admin_area/img/broadcast_messages.png +++ b/doc/user/admin_area/img/broadcast_messages.png diff --git a/doc/user/admin_area/img/license_details.png b/doc/user/admin_area/img/license_details.png Binary files differindex 2085bb437ad..3e980d9316d 100644 --- a/doc/user/admin_area/img/license_details.png +++ b/doc/user/admin_area/img/license_details.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index f5e6bff67c5..9bc0f64b68d 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -272,7 +272,7 @@ The **Logs** page provides access to the following log files: | Log file | Contents | | :---------------------- | :------- | | `application.log` | GitLab user activity | -| `githost.log` | Failed GitLab interaction with Git repositories | +| `git_json.log` | Failed GitLab interaction with Git repositories | | `production.log` | Requests received from Unicorn, and the actions taken to serve those requests | | `sidekiq.log` | Background jobs | | `repocheck.log` | Repository activity | diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index bbd04146eb2..f5864e1f828 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -30,22 +30,22 @@ Otherwise, you can: 1. Navigate manually to the **Admin Area** by clicking the wrench icon in the menu bar. -  +  1. And then going to the **License** tab and click on **Upload New License**. -  +  1. If you've received a `.gitlab-license` file, you should have already downloaded it in your local machine. You can then upload it directly by choosing the license file and clicking the **Upload license** button. In the image below, you can see that the selected license file is named `GitLab.gitlab-license`. -  +  - If you've received your license as plain text, you need to select the - "Enter license key" option, copy the license, paste it into the "License key" - field and click **Upload license**. + If you've received your license as plain text, you need to select the + "Enter license key" option, copy the license, paste it into the "License key" + field and click **Upload license**. ## Add your license at install time diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index 35e7b6fb541..52f24c602df 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -2,7 +2,7 @@ type: concepts, howto --- -# Health Check +# Health Check **(CORE ONLY)** > - Liveness and readiness probes were [introduced][ce-10416] in GitLab 9.1. > - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and was @@ -21,28 +21,63 @@ traffic until the system is ready or restart the container as needed. To access monitoring resources, the requesting client IP needs to be included in a whitelist. For details, see [how to add IPs to a whitelist for the monitoring endpoints](../../../administration/monitoring/ip_whitelist.md). -## Using the endpoints +## Using the endpoints locally With default whitelist settings, the probes can be accessed from localhost using the following URLs: -- `http://localhost/-/health` -- `http://localhost/-/readiness` -- `http://localhost/-/liveness` +```text +GET http://localhost/-/health +``` + +```text +GET http://localhost/-/readiness +``` + +```text +GET http://localhost/-/liveness +``` + +## Health -The first endpoint, `health`, only checks whether the application server is running. It does not verify the database or other services are running. A successful response will return a 200 status code with the following message: +Checks whether the application server is running. It does not verify the database or other services are running. + +```text +GET /-/health +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/health' +``` + +Example response: ```text GitLab OK ``` -The readiness and liveness probes will provide a report of system health in JSON format. +## Readiness + +The readiness probe checks whether the Gitlab instance is ready to use. It checks the dependent services (Database, Redis, Gitaly etc.) and gives a status for each. + +```text +GET /-/readiness +``` + +Example request: -`readiness` probe example output: +```sh +curl 'https://gitlab.example.com/-/readiness' +``` + +Example response: ```json { "db_check":{ - "status":"ok" + "status":"failed", + "message": "unexpected Db check result: 0" }, "redis_check":{ "status":"ok" @@ -65,7 +100,23 @@ The readiness and liveness probes will provide a report of system health in JSON } ``` -`liveness` probe example output: +## Liveness + +The liveness probe checks whether the application server is alive. Unlike the [`health`](#health) check, this check hits the database. + +```text +GET /-/liveness +``` + +Example request: + +```sh +curl 'https://gitlab.example.com/-/liveness' +``` + +Example response: + +On success, the endpoint will return a valid successful HTTP status code, and a response like below. ```json { @@ -90,10 +141,7 @@ The readiness and liveness probes will provide a report of system health in JSON } ``` -## Status - -On failure, the endpoint will return a `500` HTTP status code. On success, the endpoint -will return a valid successful HTTP status code, and a `success` message. +On failure, the endpoint will return a `500` HTTP status code. ## Access token (Deprecated) diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 9968b7349dc..5e385b7216d 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -2,7 +2,18 @@ type: reference --- -# Account and limit settings +# Account and limit settings **(CORE ONLY)** + +## Max attachment size + +You can change the maximum file size for attachments in comments and replies in GitLab. +Navigate to **Admin Area (wrench icon) > Settings > General**, then expand **Account and Limit**. +From here, you can increase or decrease by changing the value in `Maximum attachment size (MB)`. + +NOTE: **Note:** +If you choose a size larger than what is currently configured for the web server, +you will likely get errors. See the [troubleshooting section](#troubleshooting) for more +details. ## Repository size limit **(STARTER)** @@ -41,7 +52,7 @@ These settings can be found within: - The path `/admin/application_settings`. The first push of a new project, including LFS objects, will be checked for size -and **will** be rejected if the sum of their sizes exceeds the maximum allowed +and **will** be rejected if the sum of their sizes exceeds the maximum allowed repository size. **Note:** The repository size limit includes repository files and LFS, and does not include artifacts. @@ -51,14 +62,18 @@ For details on manually purging files, see [reducing the repository size using G NOTE: **Note:** GitLab.com repository size [is set by GitLab](../../gitlab_com/index.md#repository-size-limit). -<!-- ## Troubleshooting +## Troubleshooting + +### 413 Request Entity Too Large + +If you are attaching a file to a comment or reply in GitLab and receive the `413 Request Entity Too Large` +error, it is likely caused by having a [max attachment size](#max-attachment-size) +larger than what the web server is configured to allow. -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +If you wanted to increase the max attachment size to 200m in a GitLab +[Omnibus](https://docs.gitlab.com/omnibus/) install, for example, you might need to +add the line below to `/etc/gitlab/gitlab.rb` before increasing the max attachment size: -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +``` +nginx['client_max_body_size'] = "200m" +``` diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index ebbb2472752..bd76b052422 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -94,49 +94,6 @@ a group in the **Usage Quotas** page available to the group page settings list.  -## Extra Shared Runners pipeline minutes quota **[FREE ONLY]** - -If you're using GitLab.com, you can purchase additional CI minutes so your -pipelines will not be blocked after you have used all your CI minutes from your -main quota. - -In order to purchase additional minutes, you should follow these steps: - -1. Go to **Group > Settings > Pipelines quota**. Once you are on that page, click on **Buy additional minutes**. - -  - -1. Locate the subscription card that is linked to your group on GitLab.com, - click on **Buy more CI minutes**, and complete the details about the transaction. - -  - -1. Once we have processed your payment, the extra CI minutes - will be synced to your Group and you can visualize it from the - **Group > Settings > Pipelines quota** page: - -  - -Be aware that: - -1. If you have purchased extra CI minutes before the purchase of a paid plan, - we will calculate a pro-rated charge for your paid plan. That means you may - be charged for less than one year since your subscription was previously - created with the extra CI minutes. -1. Once the extra CI minutes has been assigned to a Group they cannot be transferred - to a different Group. -1. If you have some minutes used over your default quota, these minutes will - be deducted from your Additional Minutes quota immediately after your purchase of additional - minutes. - -## What happens when my CI minutes quota run out - -When the CI minutes quota run out, an email is sent automatically to notifies the owner(s) of the group/namespace which -includes a link to [purchase more minutes](https://customers.gitlab.com/plans). - -If you are not the owner of the group, you will need to contact them to let them know they need to -[purchase more minutes](https://customers.gitlab.com/plans). - ## Archive jobs **(CORE ONLY)** Archiving jobs is useful for reducing the CI/CD footprint on the system by @@ -164,3 +121,23 @@ questions that you know someone might ask. Each scenario can be a third-level heading, e.g. `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> + +## Required pipeline configuration **(PREMIUM ONLY)** + +GitLab administrators can force a pipeline configuration to run on every +pipeline. + +The configuration applies to all pipelines for a GitLab instance and is +sourced from: + +- The [instance template repository](instance_template_repository.md). +- GitLab-supplied configuration. + +To set required pipeline configuration: + +1. Go to **Admin area > Settings > CI/CD**. +1. Expand the **Required pipeline configuration** section. +1. Select the required configuration from the provided dropdown. +1. Click **Save changes**. + + diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 1f07a4dfdc6..ddf989d0181 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -2,7 +2,7 @@ type: reference --- -# Email +# Email **(CORE ONLY)** You can customize some of the content in emails sent from your GitLab instance. @@ -17,11 +17,10 @@ The logo in the header of some emails can be customized, see the [logo customiza The additional text will appear at the bottom of any email and can be used for legal/auditing/compliance reasons. -1. Go to **Admin area > Settings** (`/admin/application_settings`). -1. Under the **Email** section, change the **Additional text** field. -1. Hit **Save** for the changes to take effect. - - +1. Go to **Admin Area > Settings > Preferences** (`/admin/application_settings/preferences`). +1. Expand the **Email** section. +1. Enter your text in the **Additional text** field. +1. Click **Save**. [ee-5031]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5031 [eep]: https://about.gitlab.com/pricing/ @@ -35,11 +34,13 @@ This configuration option sets the email hostname for [private commit emails](.. In order to change this option: -1. Go to **Admin area > Settings** (`/admin/application_settings`). -1. Under the **Email** section, change the **Custom hostname (for private commit emails)** field. -1. Hit **Save** for the changes to take effect. +1. Go to **Admin Area > Settings > Preferences** (`/admin/application_settings/preferences`). +1. Expand the **Email** section. +1. Enter the desire hostname in the **Custom hostname (for private commit emails)** field. +1. Click **Save changes**. -NOTE: **Note**: Once the hostname gets configured, every private commit email using the previous hostname, will not get +NOTE: **Note:** +Once the hostname gets configured, every private commit email using the previous hostname, will not get recognized by GitLab. This can directly conflict with certain [Push rules](../../../push_rules/push_rules.md) such as `Check whether author is a GitLab user` and `Check whether committer is the current authenticated user`. @@ -53,4 +54,4 @@ questions that you know someone might ask. Each scenario can be a third-level heading, e.g. `### Getting error message X`. If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. -->
\ No newline at end of file +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/img/additional_minutes.png b/doc/user/admin_area/settings/img/additional_minutes.png Binary files differdeleted file mode 100644 index d148ed79b92..00000000000 --- a/doc/user/admin_area/settings/img/additional_minutes.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/admin_required_pipeline.png b/doc/user/admin_area/settings/img/admin_required_pipeline.png Binary files differnew file mode 100644 index 00000000000..501b1e3ba0a --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_required_pipeline.png diff --git a/doc/user/admin_area/settings/img/buy_btn.png b/doc/user/admin_area/settings/img/buy_btn.png Binary files differdeleted file mode 100644 index 0cc88b8a48f..00000000000 --- a/doc/user/admin_area/settings/img/buy_btn.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/buy_minutes_card.png b/doc/user/admin_area/settings/img/buy_minutes_card.png Binary files differdeleted file mode 100644 index cf4ad34ead7..00000000000 --- a/doc/user/admin_area/settings/img/buy_minutes_card.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/email_confirmation.png b/doc/user/admin_area/settings/img/email_confirmation.png Binary files differnew file mode 100644 index 00000000000..987aa10c3ce --- /dev/null +++ b/doc/user/admin_area/settings/img/email_confirmation.png diff --git a/doc/user/admin_area/settings/img/email_settings.png b/doc/user/admin_area/settings/img/email_settings.png Binary files differdeleted file mode 100644 index ed0a80d10ce..00000000000 --- a/doc/user/admin_area/settings/img/email_settings.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/rate_limits_on_raw_endpoints.png b/doc/user/admin_area/settings/img/rate_limits_on_raw_endpoints.png Binary files differnew file mode 100644 index 00000000000..c32eb93c8a8 --- /dev/null +++ b/doc/user/admin_area/settings/img/rate_limits_on_raw_endpoints.png diff --git a/doc/user/admin_area/settings/img/user_and_ip_rate_limits.png b/doc/user/admin_area/settings/img/user_and_ip_rate_limits.png Binary files differnew file mode 100644 index 00000000000..53dc0e4ac87 --- /dev/null +++ b/doc/user/admin_area/settings/img/user_and_ip_rate_limits.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 5427d04cd7d..2a12614e325 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -18,6 +18,7 @@ include: - [Third party offers](third_party_offers.md) - [Usage statistics](usage_statistics.md) - [Visibility and access controls](visibility_and_access_controls.md) +- [User and IP rate limits](user_and_ip_rate_limits.md) - [Custom templates repository](instance_template_repository.md) **(PREMIUM)** NOTE: **Note:** diff --git a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md new file mode 100644 index 00000000000..8e53a6995fb --- /dev/null +++ b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md @@ -0,0 +1,22 @@ +--- +type: reference +--- + +# Rate limits on raw endpoints **(CORE ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30829) in GitLab 12.2. + +This setting allows you to rate limit the requests to raw endpoints, defaults to `300` requests per minute. +It can be modified in **Admin Area > Network > Performance Optimization**. + +For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/controllers/application_controller.rb` will be blocked. Access to the raw file will be released after 1 minute. + + + +This limit is: + +- Applied independently per project, per commit and per file path. +- Not applied per IP address. +- Active by default. To disable, set the option to `0`. + +Requests over the rate limit are logged into `auth.log`. diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index cebf36c7ec1..aea717e806d 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -2,21 +2,28 @@ type: reference --- -# Sign-up restrictions +# Sign-up restrictions **(CORE ONLY)** -You can block email addresses of specific domains, or whitelist only some -specific domains via the **Application Settings** in the Admin area. +You can use sign-up restrictions to require user email confirmation, as well as +to blacklist or whitelist email addresses belonging to specific domains. >**Note**: These restrictions are only applied during sign-up. An admin is able to add a user through the admin panel with a disallowed domain. Also note that the users can change their email addresses after signup to disallowed domains. +## Require email confirmation + +You can send confirmation emails during sign-up and require that users confirm +their email address before they are allowed to sign in. + + + ## Whitelist email domains > [Introduced][ce-598] in GitLab 7.11.0 -You can restrict users to only signup using email addresses matching the given +You can restrict users to only sign up using email addresses matching the given domains list. ## Blacklist email domains @@ -24,17 +31,23 @@ domains list. > [Introduced][ce-5259] in GitLab 8.10. With this feature enabled, you can block email addresses of a specific domain -from creating an account on your GitLab server. This is particularly useful to -prevent spam. Disposable email addresses are usually used by malicious users to -create dummy accounts and spam issues. +from creating an account on your GitLab server. This is particularly useful +to prevent malicious users from creating spam accounts with disposable email +addresses. ## Settings -This feature can be activated via the **Application Settings** in the Admin area, -and you have the option of entering the list manually, or uploading a file with -the list. +To access this feature: + +1. Navigate to the **Settings > General** in the Admin area. +1. Expand the **Sign-up restrictions** section. + +For the blacklist, you can enter the list manually or upload a `.txt` file that +contains list entries. + +For the whitelist, you must enter the list manually. -Both whitelist and blacklist accept wildcards, so for example, you can use +Both the whitelist and blacklist accept wildcards. For example, you can use `*.company.com` to accept every `company.com` subdomain, or `*.io` to block all domains ending in `.io`. Domains should be separated by a whitespace, semicolon, comma, or a new line. diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index a1bce5a6c69..baf219ac9c7 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -2,7 +2,7 @@ type: reference --- -# Enforce accepting Terms of Service +# Enforce accepting Terms of Service **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18570) > in [GitLab Core](https://about.gitlab.com/pricing/) 10.8 diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index d3c9cf7d8ff..ca26147b287 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -2,7 +2,7 @@ type: reference --- -# Third party offers +# Third party offers **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20379) > in [GitLab Core](https://about.gitlab.com/pricing/) 11.1 diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index f698e0a1608..efac7e699f3 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -2,7 +2,7 @@ type: reference --- -# Usage statistics +# Usage statistics **(CORE ONLY)** GitLab Inc. will periodically collect information about your instance in order to perform various actions. @@ -52,8 +52,8 @@ You can view the exact JSON payload in the administration panel. To view the pay 1. Expand **Settings** in the left sidebar and click on **Metrics and profiling**. 1. Expand **Usage statistics** and click on the **Preview payload** button. -You can see how [the usage ping data maps to different stages of the product](https://gitlab.com/gitlab-data/analytics/blob/master/transform/snowflake-dbt/data/ping_metrics_to_stage_mapping_data.csv). - +You can see how [the usage ping data maps to different stages of the product](https://gitlab.com/gitlab-data/analytics/blob/master/transform/snowflake-dbt/data/ping_metrics_to_stage_mapping_data.csv). + ### Deactivate the usage ping The usage ping is opt-out. If you want to deactivate this feature, go to diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md new file mode 100644 index 00000000000..b9d93bf3671 --- /dev/null +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -0,0 +1,32 @@ +--- +type: reference +--- + +# User and IP rate limits **(CORE ONLY)** + +Rate limiting is a common technique used to improve the security and durability +of a web application. For more details, see +[Rate limits](../../../security/rate_limits.md). + +The following limits can be enforced in **Admin Area > Network > User and +IP rate limits**: + +- Unauthenticated requests +- Authenticated API requests +- Authenticated web requests + +These limits are disabled by default. + + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index bf59f49b993..1e2f5705728 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -2,7 +2,7 @@ type: reference --- -# Visibility and access controls +# Visibility and access controls **(CORE ONLY)** GitLab allows administrators to: diff --git a/doc/user/analytics/cycle_analytics.md b/doc/user/analytics/cycle_analytics.md new file mode 100644 index 00000000000..b7389c8689d --- /dev/null +++ b/doc/user/analytics/cycle_analytics.md @@ -0,0 +1,182 @@ +# Cycle Analytics + +> - Introduced prior to GitLab 12.2 at the project level. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2 at the group level (enabled by feature flag `analytics`). + +Cycle Analytics measures the time spent to go from an [idea to production] - also known +as cycle time - for each of your projects. Cycle Analytics displays the median time for an idea to +reach production, along with the time typically spent in each DevOps stage along the way. + +Cycle Analytics is useful in order to quickly determine the velocity of a given +project. It points to bottlenecks in the development process, enabling management +to uncover, triage, and identify the root cause of slowdowns in the software development life cycle. + +Cycle Analytics is tightly coupled with the [GitLab flow] and +calculates a separate median for each stage. + +## Overview + +Cycle Analytics is available: + +- From GitLab 12.2, at the group level in the analytics workspace at + **Analytics > Cycle Analytics**. **(PREMIUM)** + + In the future, multiple groups will be selectable which will effectively make this an + instance-level feature. + + NOTE: **Note:** + Requires the [analytics workspace](index.md) to be enabled. + +- At the project level via **Project > Cycle Analytics**. + +There are seven stages that are tracked as part of the Cycle Analytics calculations. + +- **Issue** (Tracker) + - Time to schedule an issue (by milestone or by adding it to an issue board) +- **Plan** (Board) + - Time to first commit +- **Code** (IDE) + - Time to create a merge request +- **Test** (CI) + - Time it takes GitLab CI/CD to test your code +- **Review** (Merge Request/MR) + - Time spent on code review +- **Staging** (Continuous Deployment) + - Time between merging and deploying to production +- **Production** (Total) + - Total lifecycle time; i.e. the velocity of the project or team + +## How the data is measured + +Cycle Analytics records cycle time and data based on the project issues with the +exception of the staging and production stages, where only data deployed to +production are measured. + +Specifically, if your CI is not set up and you have not defined a `production` +or `production/*` [environment], then you will not have any data for those stages. + +Each stage of Cycle Analytics is further described in the table below. + +| **Stage** | **Description** | +| --------- | --------------- | +| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list](../project/issue_board.md#creating-a-new-list) created for it. | +| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. | +| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. | +| Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. | +| Review | Measures the median time taken to review the merge request that has closing issue pattern, between its creation and until it's merged. | +| Staging | Measures the median time between merging the merge request with closing issue pattern until the very first deployment to production. It's tracked by the [environment] set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a production environment, this is not tracked. | +| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. | + +--- + +How this works, behind the scenes: + +1. Issues and merge requests are grouped together in pairs, such that for each + `<issue, merge request>` pair, the merge request has the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) + for the corresponding issue. All other issues and merge requests are **not** + considered. +1. Then the `<issue, merge request>` pairs are filtered out by last XX days (specified + by the UI - default is 90 days). So it prohibits these pairs from being considered. +1. For the remaining `<issue, merge request>` pairs, we check the information that + we need for the stages, like issue creation date, merge request merge time, + etc. + +To sum up, anything that doesn't follow [GitLab flow] will not be tracked and the +Cycle Analytics dashboard will not present any data for: + +- merge requests that do not close an issue. +- issues not labeled with a label present in the Issue Board or for issues not assigned a milestone. +- staging and production stages, if the project has no `production` or `production/*` + environment. + +## Example workflow + +Below is a simple fictional workflow of a single cycle that happens in a +single day passing through all seven stages. Note that if a stage does not have +a start and a stop mark, it is not measured and hence not calculated in the median +time. It is assumed that milestones are created and CI for testing and setting +environments is configured. + +1. Issue is created at 09:00 (start of **Issue** stage). +1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of + **Plan** stage). +1. Start working on the issue, create a branch locally and make one commit at + 12:00. +1. Make a second commit to the branch which mentions the issue number at 12.30 + (stop of **Plan** stage / start of **Code** stage). +1. Push branch and create a merge request that contains the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) + in its description at 14:00 (stop of **Code** stage / start of **Test** and + **Review** stages). +1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and + takes 5min (stop of **Test** stage). +1. Review merge request, ensure that everything is OK and merge the merge + request at 19:00. (stop of **Review** stage / start of **Staging** stage). +1. Now that the merge request is merged, a deployment to the `production` + environment starts and finishes at 19:30 (stop of **Staging** stage). +1. The cycle completes and the sum of the median times of the previous stages + is recorded to the **Production** stage. That is the time between creating an + issue and deploying its relevant merge request to production. + +From the above example you can conclude the time it took each stage to complete +as long as their total time: + +- **Issue**: 2h (11:00 - 09:00) +- **Plan**: 1h (12:00 - 11:00) +- **Code**: 2h (14:00 - 12:00) +- **Test**: 5min +- **Review**: 5h (19:00 - 14:00) +- **Staging**: 30min (19:30 - 19:00) +- **Production**: Since this stage measures the sum of median time off all + previous stages, we cannot calculate it if we don't know the status of the + stages before. In case this is the very first cycle that is run in the project, + then the **Production** time is 10h 30min (19:30 - 09:00) + +A few notes: + +- In the above example we demonstrated that it doesn't matter if your first + commit doesn't mention the issue number, you can do this later in any commit + of the branch you are working on. +- You can see that the **Test** stage is not calculated to the overall time of + the cycle since it is included in the **Review** process (every MR should be + tested). +- The example above was just **one cycle** of the seven stages. Add multiple + cycles, calculate their median time and the result is what the dashboard of + Cycle Analytics is showing. + +## Permissions + +The current permissions on the Project Cycle Analytics dashboard are: + +- Public projects - anyone can access +- Internal projects - any authenticated user can access +- Private projects - any member Guest and above can access + +You can [read more about permissions][permissions] in general. + +NOTE: **Note:** +As of GitLab 12.2, the project-level page is deprecated. You should access +project-level Cycle Analytics from **Analytics > Cycle Analytics** in the top +navigation bar. We will ensure that the same project-level functionality is available +to CE users in the new analytics space. + +For Cycle Analytics functionality introduced in GitLab 12.2 and later: + +- Users must have Reporter access or above. +- Features are available only on + [Premium or Silver tiers](https://about.gitlab.com/pricing/) and above. + +## More resources + +Learn more about Cycle Analytics in the following resources: + +- [Cycle Analytics feature page](https://about.gitlab.com/features/cycle-analytics/) +- [Cycle Analytics feature preview](https://about.gitlab.com/2016/09/16/feature-preview-introducing-cycle-analytics/) +- [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/) + +[ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986 +[ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975 +[environment]: ../../ci/yaml/README.md#environment +[GitLab flow]: ../../workflow/gitlab_flow.md +[idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab +[permissions]: ../permissions.md +[yml]: ../../ci/yaml/README.md diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md new file mode 100644 index 00000000000..ec719c0b4a1 --- /dev/null +++ b/doc/user/analytics/index.md @@ -0,0 +1,22 @@ +# Analytics workspace + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12077) in GitLab 12.2 (enabled using `analytics` feature flag). + +The Analytics workspace will make it possible to aggregate analytics across +GitLab, so that users can view information across multiple projects and groups +in one place. + +To access the centralized analytics workspace: + +1. Ensure it's enabled. Requires a GitLab administrator to enable it with the `analytics` feature + flag. +1. Once enabled, click on **Analytics** from the top navigation bar. + +## Available analytics + +From the centralized analytics workspace, the following analytics are available: + +- [Cycle Analytics](cycle_analytics.md). + +NOTE: **Note:** +Project-level Cycle Analytics are still available at a project's **Project > Cycle Analytics**. diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 696446599c8..7b631a5a1cd 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Container Scanning **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3672) @@ -10,7 +14,7 @@ images (or more precisely the containers) for known vulnerabilities by using [Clair](https://github.com/coreos/clair) and [clair-scanner](https://github.com/arminc/clair-scanner), two open source tools for Vulnerability Static Analysis for containers. -You can take advantage of Container Scanning by either [including the CI job](#including-the-provided-template) in +You can take advantage of Container Scanning by either [including the CI job](#configuration) in your existing `.gitlab-ci.yml` file or by implicitly using [Auto Container Scanning](../../../topics/autodevops/index.md#auto-container-scanning-ultimate) that is provided by [Auto DevOps](../../../topics/autodevops/index.md). @@ -47,7 +51,7 @@ To enable Container Scanning in your pipeline, you need: your Docker image to your project's [Container Registry](../../project/container_registry.md). The name of the Docker image should match the following scheme: - ``` + ```text $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA ``` @@ -55,32 +59,16 @@ To enable Container Scanning in your pipeline, you need: [predefined environment variables](../../../ci/variables/predefined_variables.md) document. -## Configuring Container Scanning - -To enable Container Scanning in your project, define a job in your -`.gitlab-ci.yml` file that generates the -[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate). - -This can be done in two ways: +## Configuration -- For GitLab 11.9 and later, including the provided - `Container-Scanning.gitlab-ci.yml` template (recommended). -- Manually specifying the job definition. Not recommended unless using GitLab - 11.8 and earlier. +For GitLab 11.9 and later, to enable Container Scanning, you must +[include](../../../ci/yaml/README.md#includetemplate) the +[`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) +that's provided as a part of your GitLab installation. +For GitLab versions earlier than 11.9, you can copy and use the job as defined +in that template. -### Including the provided template - -NOTE: **Note:** -The CI/CD Container Scanning template is supported on GitLab 11.9 and later versions. -For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). - -A CI/CD [Container Scanning template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) -with the default Container Scanning job definition is provided as a part of your GitLab -installation that you can [include](../../../ci/yaml/README.md#includetemplate) -in your `.gitlab-ci.yml` file. - -To enable Container Scanning using the provided template, add the following to -your `.gitlab-ci.yml` file: +Add the following to your `.gitlab-ci.yml` file: ```yaml include: @@ -89,12 +77,12 @@ include: The included template will: -- Create a `container_scanning` job in your CI/CD pipeline. -- Pull the already built Docker image from your project's - [Container Registry](../../project/container_registry.md) (see [requirements](#requirements)) - and scan it for possible vulnerabilities. +1. Create a `container_scanning` job in your CI/CD pipeline. +1. Pull the already built Docker image from your project's + [Container Registry](../../project/container_registry.md) (see [requirements](#requirements)) + and scan it for possible vulnerabilities. -The report will be saved as a +The results will be saved as a [Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate) that you can later download and analyze. Due to implementation limitations, we always take the latest Container Scanning @@ -106,95 +94,36 @@ If you want to whitelist some specific vulnerabilities, you can do so by definin them in a YAML file named `clair-whitelist.yml`. Read more in the [Clair documentation](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file). -### Manual job definition for GitLab 11.5 and later - -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -However, if you're using GitLab 11.9+, it's recommended to use -[the provided Container Scanning template](#including-the-provided-template). +## Example -For GitLab 11.5 and GitLab Runner 11.5 and later, the following `container_scanning` -job can be added: +The following is a sample `.gitlab-ci.yml` that will build your Docker Image, push it to the container registry and run Container Scanning. ```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/README.html#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - CLAIR_LOCAL_SCAN_VERSION: v2.0.8_fe9b059d930314b54c78f75afe265955faf4fdc1 - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:${CLAIR_LOCAL_SCAN_VERSION} - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - reports: - container_scanning: gl-container-scanning-report.json -``` +variables: + DOCKER_DRIVER: overlay2 -### Manual job definition for GitLab 11.4 and earlier (deprecated) +services: + - docker:stable-dind -CAUTION: **Deprecated:** -Before GitLab 11.5, the Container Scanning job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained, they have been deprecated -and may be removed in the next major release, GitLab 12.0. You are strongly -advised to update your current `.gitlab-ci.yml` configuration to reflect that change. +stages: + - build + - test -For GitLab 11.4 and earlier, the Container Scanning job should look like: +include: + - template: Container-Scanning.gitlab-ci.yml -```yaml -container_scanning: +build: image: docker:stable + stage: build variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/README.html#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - CLAIR_LOCAL_SCAN_VERSION: v2.0.8_fe9b059d930314b54c78f75afe265955faf4fdc1 - allow_failure: true - services: - - docker:stable-dind + IMAGE: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:${CLAIR_LOCAL_SCAN_VERSION} - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - paths: [gl-container-scanning-report.json] + - docker info + - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker build -t $IMAGE . + - docker push $IMAGE ``` -Alternatively, the job name could be `sast:container` -and the artifact name could be `gl-sast-container-report.json`. -These names have been deprecated with GitLab 11.0 -and may be removed in the next major release, GitLab 12.0. - ## Security Dashboard The Security Dashboard is a good place to get an overview of all the security @@ -219,11 +148,11 @@ When the GitLab Runner uses the Docker executor and NFS is used (e.g., `/var/lib/docker` is on an NFS mount), Container Scanning might fail with an error like the following: -``` +```text docker: Error response from daemon: failed to copy xattrs: failed to set xattr "security.selinux" on /path/to/file: operation not supported. ``` This is a result of a bug in Docker which is now [fixed](https://github.com/containerd/continuity/pull/138 "fs: add WithAllowXAttrErrors CopyOpt"). To prevent the error, ensure the Docker version that the Runner is using is `18.09.03` or higher. For more information, see -[issue #10241](https://gitlab.com/gitlab-org/gitlab-ee/issues/10241 "Investigate why Container Scanning is not working with NFS mounts"). +[issue #10241](https://gitlab.com/gitlab-org/gitlab-ee/issues/10241 "Investigate why Container Scanning is not working with NFS mounts").
\ No newline at end of file diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 936703cce32..fa84f995e58 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -1,8 +1,17 @@ +--- +type: reference, howto +--- + # Dynamic Application Security Testing (DAST) **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4348) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. +NOTE: **4 of the top 6 attacks were application based.** +Download our whitepaper, +["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) +to learn how to protect your organization. + Running [static checks](../sast/index.md) on your code is the first step to detect vulnerabilities that can put the security of your code at risk. Yet, once deployed, your application is exposed to a new category of possible attacks, @@ -14,7 +23,7 @@ Dynamic Application Security Testing (DAST) comes into place. If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your running web application(s) for known vulnerabilities using Dynamic Application Security Testing (DAST). -You can take advantage of DAST by either [including the CI job](#configuring-dast) in +You can take advantage of DAST by either [including the CI job](#configuration) in your existing `.gitlab-ci.yml` file or by implicitly using [Auto DAST](../../../topics/autodevops/index.md#auto-dast-ultimate) that is provided by [Auto DevOps](../../../topics/autodevops/index.md). @@ -51,30 +60,16 @@ applications while you are developing and testing your applications. To run a DAST job, you need GitLab Runner with the [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). -## Configuring DAST - -To enable DAST in your project, define a job in your `.gitlab-ci.yml` file that generates the -[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate). - -This can be done in two ways: - -- For GitLab 11.9 and later, including the provided `DAST.gitlab-ci.yml` template (recommended). -- Manually specifying the job definition. Not recommended unless using GitLab - 11.8 and earlier. - -### Including the provided template +## Configuration -NOTE: **Note:** -The CI/CD DAST template is supported on GitLab 11.9 and later versions. -For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). +For GitLab 11.9 and later, to enable DAST, you must +[include](../../../ci/yaml/README.md#includetemplate) the +[`DAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml) +that's provided as a part of your GitLab installation. +For GitLab versions earlier than 11.9, you can copy and use the job as defined +in that template. -A CI/CD [DAST template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml) -with the default DAST job definition is provided as a part of your GitLab -installation which you can [include](../../../ci/yaml/README.md#includetemplate) -in your `.gitlab-ci.yml` file. - -To enable DAST using the provided template, add the following to your `.gitlab-ci.yml` -file: +Add the following to your `.gitlab-ci.yml` file: ```yaml include: @@ -84,22 +79,22 @@ variables: DAST_WEBSITE: https://example.com ``` +There are two ways to define the URL to be scanned by DAST: + +- Set the `DAST_WEBSITE` [variable](../../../ci/yaml/README.md#variables). +- Add it in an `environment_url.txt` file at the root of your project. + The included template will create a `dast` job in your CI/CD pipeline and scan your project's source code for possible vulnerabilities. -The report will be saved as a +The results will be saved as a [DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate) that you can later download and analyze. Due to implementation limitations we always take the latest DAST artifact available. Behind the scenes, the [GitLab DAST Docker image](https://gitlab.com/gitlab-org/security-products/dast) is used to run the tests on the specified URL and scan it for possible vulnerabilities. -There are two ways to define the URL to be scanned by DAST: - -- Set the `DAST_WEBSITE` [variable](../../../ci/yaml/README.md#variables). -- Add it in an `environment_url.txt` file at the root of your project. - -#### Authenticated scan +### Authenticated scan It's also possible to authenticate the user before performing the DAST checks: @@ -117,12 +112,12 @@ variables: DAST_AUTH_EXCLUDE_URLS: http://example.com/sign-out,http://example.com/sign-out-2 # optional, URLs to skip during the authenticated scan; comma-separated, no spaces in between ``` -The report will be saved as a +The results will be saved as a [DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate) that you can later download and analyze. Due to implementation limitations, we always take the latest DAST artifact available. -#### Full scan +### Full scan DAST can be configured to perform [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan), which includes both passive and active scanning against the same target website: @@ -135,7 +130,7 @@ variables: DAST_FULL_SCAN_ENABLED: "true" ``` -#### Customizing the DAST settings +### Customizing the DAST settings The DAST settings can be changed through environment variables by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. @@ -155,7 +150,7 @@ variables: Because the template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline configuration, the last mention of the variable will take precedence. -#### Overriding the DAST template +### Overriding the DAST template If you want to override the job definition (for example, change properties like `variables` or `dependencies`), you need to declare a `dast` job after the @@ -176,78 +171,27 @@ As the DAST job belongs to a separate `dast` stage that runs after all [default stages](../../../ci/yaml/README.md#stages), don't forget to add `stage: dast` when you override the template job definition. -### Manual job definition for GitLab 11.5 and later - -For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dast` -job can be added: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` - -Where the `website` variable holds the URL to run the tests against. - -For an authenticated scan, use the following definition: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - login_url: "https://example.com/sign-in" - username: "john.doe@example.com" - password: "john-doe-password" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website - --auth-url $login_url - --auth-username $username - --auth-password $password || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` - -See the [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) -to learn more about the authentication settings. - -### Manual job definition for GitLab 11.4 and earlier (deprecated) - -CAUTION: **Caution:** -Before GitLab 11.5, DAST job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. You are strongly advised -to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - paths: [gl-dast-report.json] -``` +## Available variables + +DAST can be [configured](#customizing-the-dast-settings) using environment variables. +Since it's a wrapper around the ZAP scanning scripts +([baseline](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan) +or [full](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) scan), it +accepts all arguments those scripts recognize (the arguments are the same). +The choice of the scan type depends on the `DAST_FULL_SCAN_ENABLED` environment +variable value. + +| Environment variable | Required | Description | +|-----------------------------| ----------|--------------------------------------------------------------------------------| +| `DAST_WEBSITE` | yes | The URL of the website to scan. | +| `DAST_AUTH_URL` | no | The authentication URL of the website to scan. | +| `DAST_USERNAME` | no | The username to authenticate to in the website. | +| `DAST_PASSWORD` | no | The password to authenticate to in the website. | +| `DAST_USERNAME_FIELD` | no | The name of username field at the sign-in HTML form. | +| `DAST_PASSWORD_FIELD` | no | The name of password field at the sign-in HTML form. | +| `DAST_AUTH_EXCLUDE_URLS` | no | The URLs to skip during the authenticated scan; comma-separated, no spaces in between. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` | no | Time limit in seconds to wait for target availability. Scan is attempted nevertheless if it runs out. Integer. Defaults to `60`. | +| `DAST_FULL_SCAN_ENABLED` | no | Switches the tool to execute [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Boolean. `true`, `True`, or `1` are considered as true value, otherwise false. Defaults to `false`. | ## Security Dashboard @@ -264,3 +208,15 @@ Once a vulnerability is found, you can interact with it. Read more on how to For more information about the vulnerabilities database update, check the [maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/dependency_list/img/dependency_list_v12_2.png b/doc/user/application_security/dependency_list/img/dependency_list_v12_2.png Binary files differnew file mode 100644 index 00000000000..af9cee08d71 --- /dev/null +++ b/doc/user/application_security/dependency_list/img/dependency_list_v12_2.png diff --git a/doc/user/application_security/dependency_list/index.md b/doc/user/application_security/dependency_list/index.md new file mode 100644 index 00000000000..38c38bbd8a9 --- /dev/null +++ b/doc/user/application_security/dependency_list/index.md @@ -0,0 +1,49 @@ +# Dependency List **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + +The Dependency list allows you to see your project's dependencies, and key +details about them, including their known vulnerabilities. To see it, +navigate to **Security & Compliance > Dependency List** in your project's +sidebar. + +## Requirements + +1. The [Dependency Scanning](../dependency_scanning/index.md) CI job must be + configured for your project. +1. Your project uses at least one of the + [languages and package managers](../dependency_scanning/index.md#supported-languages-and-package-managers) + supported by Gemnasium. + +## Viewing dependencies + + + +Dependencies are displayed with the following information: + +| Field | Description | +| --------- | ----------- | +| Status | Displays whether or not the dependency has any known vulnerabilities | +| Component | The dependency's name | +| Version | The exact locked version of the dependency your project uses | +| Packager | The packager used to install the depedency | +| Location | A link to the packager-specific lockfile in your project that declared the dependency | + +Dependencies shown are initially sorted by their names. They can also be sorted +by the packager they were installed by, or by the severity of their known +vulnerabilities. + +There is a second list under the `Vulnerable components` tab displaying only +those dependencies with known vulnerabilities. If there are none, this tab is +disabled. + +### Vulnerabilities + +If a dependency has known vulnerabilities, they can be viewed by clicking on the +`Status` cell of that dependency. The severity and description of each +vulnerability will then be displayed below it. + +## Downloading the Dependency List + +Your project's full list of dependencies and their details can be downloaded in +`JSON` format by clicking on the download button. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 2fe8a6f9029..b40392e12d5 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Dependency Scanning **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5105) @@ -8,7 +12,7 @@ in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.7. If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your dependencies for known vulnerabilities using Dependency Scanning. -You can take advantage of Dependency Scanning by either [including the CI job](#including-the-provided-template) +You can take advantage of Dependency Scanning by either [including the CI job](#configuration) in your existing `.gitlab-ci.yml` file or by implicitly using [Auto Dependency Scanning](../../../topics/autodevops/index.md#auto-dependency-scanning-ultimate) that is provided by [Auto DevOps](../../../topics/autodevops/index.md). @@ -48,14 +52,15 @@ The following languages and dependency managers are supported. | Language (package managers) | Supported | Scan tool(s) | |----------------------------- | --------- | ------------ | +| Java ([Gradle](https://gradle.org/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/13075 "Dependency Scanning for Gradle" )) | not available | +| Java ([Maven](https://maven.apache.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | | JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [Retire.js](https://retirejs.github.io/retire.js) | +| Go ([Golang](https://golang.org/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7132 "Dependency Scanning for Go")) | not available | +| PHP ([Composer](https://getcomposer.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | | Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | +| Python ([Pipfile](https://docs.pipenv.org/en/latest/basics/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/11756 "Pipfile.lock support for Dependency Scanning"))| not available | +| Python ([poetry](https://poetry.eustace.io/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7006 "Support Poetry in Dependency Scanning")) | not available | | Ruby ([gem](https://rubygems.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium), [bundler-audit](https://github.com/rubysec/bundler-audit) | -| Java ([Maven](https://maven.apache.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| PHP ([Composer](https://getcomposer.org/)) | yes | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium) | -| Python ([poetry](https://poetry.eustace.io/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7006 "Support Poetry in Dependency Scanning")) | not available | -| Python ([Pipfile](https://docs.pipenv.org/en/latest/basics/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/11756 "Pipfile.lock support for Dependency Scanning"))| not available | -| Go ([Golang](https://golang.org/)) | no ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7132 "Dependency Scanning for Go")) | not available | ## Remote checks @@ -74,31 +79,16 @@ The Gemnasium client does **NOT** send the exact package versions your project r You can disable the remote checks by [using](#customizing-the-dependency-scanning-settings) the `DS_DISABLE_REMOTE_CHECKS` environment variable and setting it to `true`. -## Configuring Dependency Scanning - -To enable Dependency Scanning in your project, define a job in your `.gitlab-ci.yml` -file that generates the -[Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate). - -This can be done in two ways: - -- For GitLab 11.9 and later, including the provided `Dependency-Scanning.gitlab-ci.yml` template (recommended). -- Manually specifying the job definition. Not recommended unless using GitLab - 11.8 and earlier. +## Configuration -### Including the provided template +For GitLab 11.9 and later, to enable Dependency Scanning, you must +[include](../../../ci/yaml/README.md#includetemplate) the +[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +that's provided as a part of your GitLab installation. +For GitLab versions earlier than 11.9, you can copy and use the job as defined +that template. -NOTE: **Note:** -The CI/CD Dependency Scanning template is supported on GitLab 11.9 and later versions. -For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). - -A CI/CD [Dependency Scanning template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) -with the default Dependency Scanning job definition is provided as a part of your GitLab -installation which you can [include](../../../ci/yaml/README.md#includetemplate) -in your `.gitlab-ci.yml` file. - -To enable Dependency Scanning using the provided template, add the following to -your `.gitlab-ci.yml` file: +Add the following to your `.gitlab-ci.yml` file: ```yaml include: @@ -108,12 +98,12 @@ include: The included template will create a `dependency_scanning` job in your CI/CD pipeline and scan your project's source code for possible vulnerabilities. -The report will be saved as a +The results will be saved as a [Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate) that you can later download and analyze. Due to implementation limitations, we always take the latest Dependency Scanning artifact available. -#### Customizing the Dependency Scanning settings +### Customizing the Dependency Scanning settings The Dependency Scanning settings can be changed through [environment variables](#available-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. @@ -131,7 +121,7 @@ variables: Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline configuration, the last mention of the variable will take precedence. -#### Overriding the Dependency Scanning template +### Overriding the Dependency Scanning template If you want to override the job definition (for example, change properties like `variables` or `dependencies`), you need to declare a `dependency_scanning` job @@ -146,99 +136,27 @@ dependency_scanning: CI_DEBUG_TRACE: "true" ``` -#### Available variables +### Available variables Dependency Scanning can be [configured](#customizing-the-dependency-scanning-settings) using environment variables. -| Environment variable | Function | -|-------------------------------- |----------| -| `DS_ANALYZER_IMAGES` | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). | -| `DS_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | -| `DS_ANALYZER_IMAGE_TAG` | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). | -| `DS_DEFAULT_ANALYZERS` | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). | -| `DS_DISABLE_REMOTE_CHECKS` | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). | -| `DS_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to `0` to disable). | -| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | -| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | -| `DS_PULL_ANALYZER_IMAGE_TIMEOUT` | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | -| `DS_RUN_ANALYZER_TIMEOUT` | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | - -### Manual job definition for GitLab 11.5 and later - -For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning` -job can be added: - -```yaml -dependency_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - allow_failure: true - services: - - docker:stable-dind - script: - - export DS_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} - - | - docker run \ - --env DS_ANALYZER_IMAGES \ - --env DS_ANALYZER_IMAGE_PREFIX \ - --env DS_ANALYZER_IMAGE_TAG \ - --env DS_DEFAULT_ANALYZERS \ - --env DEP_SCAN_DISABLE_REMOTE_CHECKS \ - --env DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ - --env DS_PULL_ANALYZER_IMAGE_TIMEOUT \ - --env DS_RUN_ANALYZER_TIMEOUT \ - --volume "$PWD:/code" \ - --volume /var/run/docker.sock:/var/run/docker.sock \ - "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$DS_VERSION" /code - dependencies: [] - artifacts: - reports: - dependency_scanning: gl-dependency-scanning-report.json -``` - -You can supply many other [settings variables](#available-variables) -via `docker run --env` to customize your job execution. - -### Manual job definition for GitLab 11.4 and earlier (deprecated) - -CAUTION: **Caution:** -Before GitLab 11.5, the Dependency Scanning job and artifact had to be named specifically -to automatically extract the report data and show it in the merge request widget. -While these old job definitions are still maintained, they have been deprecated -and may be removed in the next major release, GitLab 12.0. You are strongly advised -to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -dependency_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - allow_failure: true - services: - - docker:stable-dind - script: - - export DS_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} - - | - docker run \ - --env DS_ANALYZER_IMAGES \ - --env DS_ANALYZER_IMAGE_PREFIX \ - --env DS_ANALYZER_IMAGE_TAG \ - --env DS_DEFAULT_ANALYZERS \ - --env DS_EXCLUDED_PATHS \ - --env DEP_SCAN_DISABLE_REMOTE_CHECKS \ - --env DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ - --env DS_PULL_ANALYZER_IMAGE_TIMEOUT \ - --env DS_RUN_ANALYZER_TIMEOUT \ - --volume "$PWD:/code" \ - --volume /var/run/docker.sock:/var/run/docker.sock \ - "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$DS_VERSION" /code - artifacts: - paths: [gl-dependency-scanning-report.json] -``` +| Environment variable | Description | Example usage | +|-------------------------------- |-------------| | +| `DS_ANALYZER_IMAGES` | Comma separated list of custom images. The official default images are still enabled. Read more about [customizing analyzers](analyzers.md). | | +| `DS_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | | +| `DS_ANALYZER_IMAGE_TAG` | Override the Docker tag of the official default images. Read more about [customizing analyzers](analyzers.md). | | +| `DS_PYTHON_VERSION` | Version of Python. If set to 2, dependencies are installed using Python 2.7 instead of Python 3.6. ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12296) in GitLab 12.1)| | +| `DS_PIP_DEPENDENCY_PATH` | Path to load Python pip dependencies from. ([Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12412) in GitLab 12.2) | | +| `DS_DEFAULT_ANALYZERS` | Override the names of the official default images. Read more about [customizing analyzers](analyzers.md). | | +| `DS_DISABLE_REMOTE_CHECKS` | Do not send any data to GitLab. Used in the [Gemnasium analyzer](#remote-checks). | | +| `DS_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to `0` to disable). | | +| `DS_EXCLUDED_PATHS` | Exclude vulnerabilities from output based on the paths. A comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | `DS_EXCLUDED_PATHS=doc,spec` | +| `DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | | +| `DS_PULL_ANALYZER_IMAGE_TIMEOUT` | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | | +| `DS_RUN_ANALYZER_TIMEOUT` | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. For example, `300ms`, `1.5h`, or `2h45m`. | | +| `PIP_INDEX_URL` | Base URL of Python Package Index (default `https://pypi.org/simple`). | | +| `PIP_EXTRA_INDEX_URL` | Array of [extra URLs](https://pip.pypa.io/en/stable/reference/pip_install/#cmdoption-extra-index-url) of package indexes to use in addition to `PIP_INDEX_URL`. Comma separated. | | ## Reports JSON format @@ -358,8 +276,8 @@ it highlighted: Here is the description of the report file structure nodes and their meaning. All fields are mandatory to be present in the report JSON unless stated otherwise. Presence of optional fields depends on the underlying analyzers being used. -| Report JSON node | Function | -|------------------------------------------------------|----------| +| Report JSON node | Description | +|------------------------------------------------------|-------------| | `version` | Report syntax version used to generate this JSON. | | `vulnerabilities` | Array of vulnerability objects. | | `vulnerabilities[].category` | Where this vulnerability belongs (SAST, Dependency Scanning etc.). For Dependency Scanning, it will always be `dependency_scanning`. | @@ -409,16 +327,11 @@ Once a vulnerability is found, you can interact with it. Read more on how to For more information about the vulnerabilities database update, check the [maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). -## Dependency List - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +## Dependency List **(ULTIMATE)** -An additional benefit of Dependency Scanning is the ability to get a list of your -project's dependencies with their versions. This list can be generated only for -[languages and package managers](#supported-languages-and-package-managers) -supported by Gemnasium. - -To see the generated dependency list, navigate to your project's **Project > Dependency List**. +An additional benefit of Dependency Scanning is the ability to view your +project's dependencies and their known vulnerabilities. Read more about +the [Dependency List](../dependency_list/index.md). ## Versioning and release process @@ -429,3 +342,15 @@ Please check the [Release Process documentation](https://gitlab.com/gitlab-org/s You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project to find a vulnerability in the Gemnasium database. You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/img/create_issue_with_list_hover.png b/doc/user/application_security/img/create_issue_with_list_hover.png Binary files differindex 7d70e8299f5..4c38862e68f 100644 --- a/doc/user/application_security/img/create_issue_with_list_hover.png +++ b/doc/user/application_security/img/create_issue_with_list_hover.png diff --git a/doc/user/application_security/img/dismissed_info.png b/doc/user/application_security/img/dismissed_info.png Binary files differindex 64d5cf26ed2..b4470b664d2 100644 --- a/doc/user/application_security/img/dismissed_info.png +++ b/doc/user/application_security/img/dismissed_info.png diff --git a/doc/user/application_security/img/interactive_reports.png b/doc/user/application_security/img/interactive_reports.png Binary files differindex 373b39104db..1b2ef0d3da9 100644 --- a/doc/user/application_security/img/interactive_reports.png +++ b/doc/user/application_security/img/interactive_reports.png diff --git a/doc/user/application_security/img/vulnerability_solution.png b/doc/user/application_security/img/vulnerability_solution.png Binary files differindex 7443b9b6eea..d86b89a5f99 100644 --- a/doc/user/application_security/img/vulnerability_solution.png +++ b/doc/user/application_security/img/vulnerability_solution.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 91e79f6c23b..fcd683ca2db 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -1,10 +1,22 @@ +--- +type: reference, howto +--- + # GitLab Secure **(ULTIMATE)** -Check your application for security vulnerabilities that may lead to unauthorized access, -data leaks, and denial of services. GitLab will perform static and dynamic tests on the -code of your application, looking for known flaws and report them in the merge request -so you can fix them before merging. Security teams can use dashboards to get a -high-level view on projects and groups, and start remediation processes when needed. +Check your application for security vulnerabilities that may lead to +unauthorized access, data leaks, and denial of services. + +GitLab will perform static and dynamic tests on the code of your application, +looking for known flaws and report them in the merge request so you can fix +them before merging. + +Security teams can use dashboards to get a high-level view on projects and +groups, and start remediation processes when needed. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview of application security with GitLab, see +[Security Deep Dive](https://www.youtube.com/watch?v=k4vEJnGYy84). ## Security scanning tools @@ -13,9 +25,10 @@ GitLab can scan and report any vulnerabilities found in your project. | Secure scanning tool | Description | |:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------| | [Container Scanning](container_scanning/index.md) **(ULTIMATE)** | Scan Docker containers for known vulnerabilities. | +| [Dependency List](dependency_list/index.md) **(ULTIMATE)** | View your project's dependencies and their known vulnerabilities. | | [Dependency Scanning](dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | | [Dynamic Application Security Testing (DAST)](dast/index.md) **(ULTIMATE)** | Analyze running web applications for known vulnerabilities. | -| [License Management](license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | +| [License Compliance](license_management/index.md) **(ULTIMATE)** | Search your project's dependencies for their licenses. | | [Security Dashboard](security_dashboard/index.md) **(ULTIMATE)** | View vulnerabilities in all your projects and groups. | | [Static Application Security Testing (SAST)](sast/index.md) **(ULTIMATE)** | Analyze source code for known vulnerabilities. | @@ -54,7 +67,7 @@ Each security vulnerability in the merge request report or the entry, a detailed information will pop up with different possible options: - [Dismiss vulnerability](#dismissing-a-vulnerability): Dismissing a vulnerability - will place a <s>strikethrough</s> styling on it. + will place a ~~strikethrough~~ styling on it. - [Create issue](#creating-an-issue-for-a-vulnerability): The new issue will have the title and description pre-populated with the information from the vulnerability report and will be created as [confidential](../project/issues/confidential_issues.md) by default. @@ -115,16 +128,16 @@ generated by GitLab. To apply the fix: 1. Click on the vulnerability. 1. Download and review the patch file `remediation.patch`. -2. Ensure your local project has the same commit checked out that was used to generate the patch. -3. Run `git apply remediation.patch`. -4. Verify and commit the changes to your branch. +1. Ensure your local project has the same commit checked out that was used to generate the patch. +1. Run `git apply remediation.patch`. +1. Verify and commit the changes to your branch.  #### Creating a merge request from a vulnerability > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9224) in - [GitLab Ultimate](https://about.gitlab.com/pricing) 11.9. +> [GitLab Ultimate](https://about.gitlab.com/pricing) 11.9. In certain cases, GitLab will allow you to create a merge request that will automatically remediate the vulnerability. Any vulnerability that has a @@ -135,3 +148,47 @@ If this action is available there will be a **Create merge request** button in t Clicking on this button will create a merge request to apply the solution onto the source branch.  + +## Security approvals in merge requests **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9928) in [GitLab Ultimate](https://about.gitlab.com/pricing) 12.2. + +Merge Request Approvals can be configured to require approval from a member +of your security team when a vulnerability would be introduced by a merge request. + +This threshold is defined as `high`, `critical`, or `unknown` +severity. When any vulnerabilities are present within a merge request, an +approval will be required from the `Vulnerability-Check` approver group. + +### Enabling Security Approvals within a project + +To enable Security Approvals, a [project approval rule](../project/merge_requests/merge_request_approvals.md#multiple-approval-rules-premium) +must be created with the case-sensitive name `Vulnerability-Check`. This approval +group must be set with an "Approvals required" count greater than zero. + +Once this group has been added to your project, the approval rule will be enabled +for all Merge Requests. + +Any code changes made will cause the count of approvals required to reset. + +An approval will be required when a security report: + +- Contains a new vulnerability of `high`, `critical`, or `unknown` severity. +- Is not generated during pipeline execution. + +An approval will be optional when a security report: + +- Contains no new vulnerabilities. +- Contains only new vulnerabilities of `low` or `medium` severity. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/license_management/img/license_management_add_license.png b/doc/user/application_security/license_management/img/license_management_add_license.png Binary files differindex 1e1a698515b..c9a5dc14c57 100644 --- a/doc/user/application_security/license_management/img/license_management_add_license.png +++ b/doc/user/application_security/license_management/img/license_management_add_license.png diff --git a/doc/user/application_security/license_management/img/license_management_decision.png b/doc/user/application_security/license_management/img/license_management_decision.png Binary files differindex 0763130c375..fbf90bec7fd 100644 --- a/doc/user/application_security/license_management/img/license_management_decision.png +++ b/doc/user/application_security/license_management/img/license_management_decision.png diff --git a/doc/user/application_security/license_management/img/license_management_search.png b/doc/user/application_security/license_management/img/license_management_search.png Binary files differindex 7b6006cef9d..b3ffd8d95a1 100644 --- a/doc/user/application_security/license_management/img/license_management_search.png +++ b/doc/user/application_security/license_management/img/license_management_search.png diff --git a/doc/user/application_security/license_management/img/license_management_settings.png b/doc/user/application_security/license_management/img/license_management_settings.png Binary files differindex 1a2bfa78a03..2e3e8888e93 100644 --- a/doc/user/application_security/license_management/img/license_management_settings.png +++ b/doc/user/application_security/license_management/img/license_management_settings.png diff --git a/doc/user/application_security/license_management/index.md b/doc/user/application_security/license_management/index.md index 8eb231f8359..44b2671930e 100644 --- a/doc/user/application_security/license_management/index.md +++ b/doc/user/application_security/license_management/index.md @@ -1,4 +1,8 @@ -# License Management **(ULTIMATE)** +--- +type: reference, howto +--- + +# License Compliance **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5483) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. @@ -6,18 +10,18 @@ in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. ## Overview If you are using [GitLab CI/CD](../../../ci/README.md), you can search your project dependencies for their licenses -using License Management. +using License Compliance. -You can take advantage of License Management by either [including the job](#configuring-license-management) +You can take advantage of License Compliance by either [including the job](#configuration) in your existing `.gitlab-ci.yml` file or by implicitly using -[Auto License Management](../../../topics/autodevops/index.md#auto-license-management-ultimate) +[Auto License Compliance](../../../topics/autodevops/index.md#auto-license-compliance-ultimate) that is provided by [Auto DevOps](../../../topics/autodevops/index.md). -GitLab checks the License Management report, compares the licenses between the +GitLab checks the License Compliance report, compares the licenses between the source and target branches, and shows the information right on the merge request. Blacklisted licenses will be clearly visible with an `x` red icon next to them as well as new licenses which need a decision from you. In addition, you can -[manually approve or blacklist](#project-policies-for-license-management) +[manually approve or blacklist](#project-policies-for-license-compliance) licenses in your project's settings. NOTE: **Note:** @@ -27,7 +31,7 @@ will be displayed in the merge request area. That is the case when you add the Consecutive merge requests will have something to compare to and the license management report will be shown properly. - + If you are a project or group Maintainer, you can click on a license to be given the choice to approve it or blacklist it. @@ -62,36 +66,19 @@ The following languages and package managers are supported. ## Requirements -To run a License Management scanning job, you need GitLab Runner with the +To run a License Compliance scanning job, you need GitLab Runner with the [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). -## Configuring License Management - -To enable License Management in your project, define a job in your `.gitlab-ci.yml` -file that generates the [License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate). - -This can be done in two ways: +## Configuration -- For GitLab 11.9 and later, including the provided `License-Management.gitlab-ci.yml` template (recommended). -- Manually specifying the job definition. Not recommended unless using GitLab - 11.8 and earlier. +For GitLab 11.9 and later, to enable License Compliance, you must +[include](../../../ci/yaml/README.md#includetemplate) the +[`License-Management.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml) +that's provided as a part of your GitLab installation. +For GitLab versions earlier than 11.9, you can copy and use the job as defined +that template. -The License Management settings can be changed through environment variables by using the -[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. These variables are documented in the [License Management documentation](https://gitlab.com/gitlab-org/security-products/license-management#settings). - -### Including the provided template - -NOTE: **Note:** -The CI/CD License Management template is supported on GitLab 11.9 and later versions. -For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). - -A CI/CD [License Management template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml) -with the default License Management job definition is provided as a part of your GitLab -installation which you can [include](../../../ci/yaml/README.md#includetemplate) -in your `.gitlab-ci.yml` file. - -To enable License Management using the provided template, add the following to -your `.gitlab-ci.yml` file: +Add the following to your `.gitlab-ci.yml` file: ```yaml include: @@ -101,14 +88,17 @@ include: The included template will create a `license_management` job in your CI/CD pipeline and scan your dependencies to find their licenses. -The report will be saved as a -[License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) +The results will be saved as a +[License Compliance report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) that you can later download and analyze. Due to implementation limitations, we -always take the latest License Management artifact available. Behind the scenes, the -[GitLab License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +always take the latest License Compliance artifact available. Behind the scenes, the +[GitLab License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/license-management) is used to detect the languages/frameworks and in turn analyzes the licenses. -#### Installing custom dependencies +The License Compliance settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. These variables are documented in the [License Compliance documentation](https://gitlab.com/gitlab-org/security-products/license-management#settings). + +### Installing custom dependencies > Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. @@ -136,7 +126,7 @@ variables: In this example, `my-custom-install-script.sh` is a shell script at the root directory of your project. -#### Overriding the template +### Overriding the template If you want to override the job definition (for example, change properties like `variables` or `dependencies`), you need to declare a `license_management` job @@ -151,9 +141,9 @@ license_management: CI_DEBUG_TRACE: "true" ``` -#### Configuring Maven projects +### Configuring Maven projects -The License Management tool provides a `MAVEN_CLI_OPTS` environment variable which can hold +The License Compliance tool provides a `MAVEN_CLI_OPTS` environment variable which can hold the command line arguments to pass to the `mvn install` command which is executed under the hood. Feel free to use it for the customization of Maven execution. For example: @@ -179,9 +169,9 @@ If you still need to run tests during `mvn install`, add `-DskipTests=false` to > [Introduced](https://gitlab.com/gitlab-org/security-products/license-management/merge_requests/36) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. -License Management uses Python 2.7 and pip 10.0 by default. -If your project requires Python 3, you can switch to Python 3.5 and pip 19.1 -by setting the `LM_PYTHON_VERSION` environment variable to `3`. +License Compliance uses Python 3.5 and pip 19.1 by default. +If your project requires Python 2, you can switch to Python 2.7 and pip 10.0 +by setting the `LM_PYTHON_VERSION` environment variable to `2`. ```yaml include: @@ -189,71 +179,10 @@ include: license_management: variables: - LM_PYTHON_VERSION: 3 -``` - -### Manual job definition for GitLab 11.5 and later - -For GitLab 11.5 and GitLab Runner 11.5 and later, the following `license_management` -job can be added: - -```yaml -license_management: - image: - name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" - entrypoint: [""] - stage: test - allow_failure: true - script: - - /run.sh analyze . - artifacts: - reports: - license_management: gl-license-management-report.json -``` - -If you want to install custom project dependencies via the `SETUP_CMD` variable: - -```yaml -license_management: - image: - name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" - entrypoint: [""] - stage: test - variables: - SETUP_CMD: ./my-custom-install-script.sh - allow_failure: true - script: - - /run.sh analyze . - artifacts: - reports: - license_management: gl-license-management-report.json -``` - -### Manual job definition for GitLab 11.4 and earlier (deprecated) - -CAUTION: **Caution:** -Before GitLab 11.5, the License Management job and artifact had to be named specifically -to automatically extract the report data and show it in the merge request widget. -While these old job definitions are still maintained, they have been deprecated -and may be removed in the next major release, GitLab 12.0. You are strongly advised -to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -license_management: - image: - name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" - entrypoint: [""] - stage: test - allow_failure: true - script: - - /run.sh analyze . - artifacts: - paths: [gl-license-management-report.json] + LM_PYTHON_VERSION: 2 ``` -## Project policies for License Management +## Project policies for License Compliance > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5940) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. @@ -267,35 +196,32 @@ To approve or blacklist a license: 1. Either use the **Manage licenses** button in the merge request widget, or navigate to the project's **Settings > CI/CD** and expand the - **License Management** section. + **License Compliance** section. 1. Click the **Add a license** button. -  +  + 1. In the **License name** dropdown, either: - - Select one of the available licenses. You can search for licenses in the field - at the top of the list. - - Enter arbitrary text in the field at the top of the list. This will cause the text to be - added as a license name to the list. + - Select one of the available licenses. You can search for licenses in the field + at the top of the list. + - Enter arbitrary text in the field at the top of the list. This will cause the text to be + added as a license name to the list. 1. Select the **Approve** or **Blacklist** radio button to approve or blacklist respectively the selected license. - - To modify an existing license: -1. In the **License Management** list, click the **Approved/Declined** dropdown to change it to the desired status. +1. In the **License Compliance** list, click the **Approved/Declined** dropdown to change it to the desired status. -  +  Searching for Licenses: 1. Use the **Search** box to search for a specific license. -  - +  - -## License Management report under pipelines +## License Compliance report under pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5491) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2. @@ -304,4 +230,16 @@ From your project's left sidebar, navigate to **CI/CD > Pipelines** and click on pipeline ID that has a `license_management` job to see the Licenses tab with the listed licenses (if any). - + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md new file mode 100644 index 00000000000..a1bd00f34e3 --- /dev/null +++ b/doc/user/application_security/sast/analyzers.md @@ -0,0 +1,144 @@ +--- +table_display_block: true +--- + +# SAST Analyzers **(ULTIMATE)** + +SAST relies on underlying third party tools that are wrapped into what we call +"Analyzers". An analyzer is a +[dedicated project](https://gitlab.com/gitlab-org/security-products/analyzers) +that wraps a particular tool to: + +- Expose its detection logic. +- Handle its execution. +- Convert its output to the common format. + +This is achieved by implementing the [common API](https://gitlab.com/gitlab-org/security-products/analyzers/common). + +SAST supports the following official analyzers: + +- [`bandit`](https://gitlab.com/gitlab-org/security-products/analyzers/bandit) (Bandit) +- [`brakeman`](https://gitlab.com/gitlab-org/security-products/analyzers/brakeman) (Brakeman) +- [`eslint`](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) (ESLint (Javascript)) +- [`flawfinder`](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder) (Flawfinder) +- [`gosec`](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) (Gosec) +- [`nodejs-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan) (NodeJsScan) +- [`phpcs-security-audit`](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit) (PHP CS security-audit) +- [`pmd-apex`](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex) (PMD (Apex only)) +- [`secrets`](https://gitlab.com/gitlab-org/security-products/analyzers/secrets) (Secrets (Gitleaks, TruffleHog & Diffence secret detectors)) +- [`security-code-scan`](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan) (Security Code Scan (.NET)) +- [`sobelow`](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow) (Sobelow (Elixir Phoenix)) +- [`spotbugs`](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) (SpotBugs with the Find Sec Bugs plugin (Ant, Gradle and wrapper, Grails, Maven and wrapper, SBT)) +- [`tslint`](https://gitlab.com/gitlab-org/security-products/analyzers/tslint) (TSLint (Typescript)) + +The analyzers are published as Docker images that SAST will use to launch +dedicated containers for each analysis. + +SAST is pre-configured with a set of **default images** that are maintained by +GitLab, but users can also integrate their own **custom images**. + +## Official default analyzers + +Any custom change to the official analyzers can be achieved by using an +[environment variable in your `.gitlab-ci.yml`](index.md#customizing-the-sast-settings). + +### Using a custom Docker mirror + +You can switch to a custom Docker registry that provides the official analyzer +images under a different prefix. For instance, the following instructs +SAST to pull `my-docker-registry/gl-images/bandit` +instead of `registry.gitlab.com/gitlab-org/security-products/analyzers/bandit`. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: SAST.gitlab-ci.yml + +variables: + SAST_ANALYZER_IMAGE_PREFIX: my-docker-registry/gl-images +``` + +This configuration requires that your custom registry provides images for all +the official analyzers. + +### Selecting specific analyzers + +You can select the official analyzers you want to run. Here's how to enable +`bandit` and `flawfinder` while disabling all the other default ones. +In `.gitlab-ci.yml` define: + +```yaml +include: + template: SAST.gitlab-ci.yml + +variables: + SAST_DEFAULT_ANALYZERS: "bandit,flawfinder" +``` + +`bandit` runs first. When merging the reports, SAST will +remove the duplicates and will keep the `bandit` entries. + +### Disabling default analyzers + +Setting `SAST_DEFAULT_ANALYZERS` to an empty string will disable all the official +default analyzers. In `.gitlab-ci.yml` define: + +```yaml +include: + template: SAST.gitlab-ci.yml + +variables: + SAST_DEFAULT_ANALYZERS: "" +``` + +That's needed when one totally relies on [custom analyzers](#custom-analyzers). + +## Custom Analyzers + +You can provide your own analyzers as a comma separated list of Docker images. +Here's how to add `analyzers/csharp` and `analyzers/perl` to the default images: +In `.gitlab-ci.yml` define: + +```yaml +include: + template: SAST.gitlab-ci.yml + +variables: + SAST_ANALYZER_IMAGES: "my-docker-registry/analyzers/csharp,amy-docker-registry/analyzers/perl" +``` + +The values must be the full path to the container registry images, +like what you would feed to the `docker pull` command. + +NOTE: **Note:** +This configuration doesn't benefit from the integrated detection step. +SAST has to fetch and spawn each Docker image to establish whether the +custom analyzer can scan the source code. + +## Analyzers Data + +| Property \ Tool | Apex | Bandit | Brakeman | ESLint security | Find Sec Bugs | Flawfinder | Go AST Scanner | NodeJsScan | Php CS Security Audit | Security code Scan (.NET) | TSLint Security | Sobelow | +| --------------------------------------- | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :---------------------: | :-------------------------: | :-------------: | :----------------: | +| Severity | ✓ | ✓ | 𐄂 | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | 𐄂 | +| Title | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Description | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | +| File | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| Start line | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +| End line | ✓ | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | +| Start column | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | +| End column | ✓ | 𐄂 | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | 𐄂 | +| External id (e.g. CVE) | 𐄂 | 𐄂 | ⚠ | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| URLs | ✓ | 𐄂 | ✓ | 𐄂 | ⚠ | 𐄂 | ⚠ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Internal doc/explanation | ✓ | ⚠ | ✓ | 𐄂 | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | +| Solution | ✓ | 𐄂 | 𐄂 | 𐄂 | ⚠ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Confidence | 𐄂 | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | ✓ | +| Affected item (e.g. class or package) | ✓ | 𐄂 | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Source code extract | 𐄂 | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | 𐄂 | 𐄂 | 𐄂 | 𐄂 | 𐄂 | +| Internal ID | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 𐄂 | ✓ | ✓ | ✓ | ✓ | + +- ✓ => we have that data +- ⚠ => we have that data but it's partially reliable, or we need to extract it from unstructured content +- 𐄂 => we don't have that data or it would need to develop specific or inefficient/unreliable logic to obtain it. + +The values provided by these tools are heterogeneous so they are sometimes +normalized into common values (e.g., `severity`, `confidence`, etc). diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 84b45cbe6e6..3eead6ccd3f 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Static Application Security Testing (SAST) **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3775) @@ -13,7 +17,7 @@ to learn how to protect your organization. If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your source code for known vulnerabilities using Static Application Security Testing (SAST). -You can take advantage of SAST by either [including the CI job](#configuring-sast) in +You can take advantage of SAST by either [including the CI job](#configuration) in your existing `.gitlab-ci.yml` file or by implicitly using [Auto SAST](../../../topics/autodevops/index.md#auto-sast-ultimate) that is provided by [Auto DevOps](../../../topics/autodevops/index.md). @@ -55,6 +59,7 @@ The following table shows which languages, package managers and frameworks are s |-----------------------------------------------------------------------------|----------------------------------------------------------------------------------------|------------------------------| | .NET | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | | Any | [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) | 11.9 | +| Apex (Salesforce) | [pmd](https://pmd.github.io/pmd/index.html) | 12.1 | | C/C++ | [Flawfinder](https://www.dwheeler.com/flawfinder/) | 10.7 | | Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.10 | | Go | [Gosec](https://github.com/securego/gosec) | 10.7 | @@ -73,30 +78,16 @@ The Java analyzers can also be used for variants like the [Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), [Grails](https://grails.org/) and the [Maven wrapper](https://github.com/takari/maven-wrapper). -## Configuring SAST - -To enable SAST in your project, define a job in your `.gitlab-ci.yml` file that generates the -[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate). - -This can be done in two ways: - -- For GitLab 11.9 and later, including the provided `SAST.gitlab-ci.yml` template (recommended). -- Manually specifying the job definition. Not recommended unless using GitLab - 11.8 and earlier. - -### Including the provided template - -NOTE: **Note:** -The CI/CD SAST template is supported on GitLab 11.9 and later versions. -For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). +## Configuration -A CI/CD [SAST template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) -with the default SAST job definition is provided as a part of your GitLab -installation which you can [include](../../../ci/yaml/README.md#includetemplate) -in your `.gitlab-ci.yml` file. +For GitLab 11.9 and later, to enable SAST, you must +[include](../../../ci/yaml/README.md#includetemplate) the +[`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) +that's provided as a part of your GitLab installation. +For GitLab versions earlier than 11.9, you can copy and use the job as defined +that template. -To enable SAST using the provided template, add the following to your `.gitlab-ci.yml` -file: +Add the following to your `.gitlab-ci.yml` file: ```yaml include: @@ -106,14 +97,14 @@ include: The included template will create a `sast` job in your CI/CD pipeline and scan your project's source code for possible vulnerabilities. -The report will be saved as a +The results will be saved as a [SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate) that you can later download and analyze. Due to implementation limitations, we always take the latest SAST artifact available. Behind the scenes, the [GitLab SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) is used to detect the languages/frameworks and in turn runs the matching scan tools. -#### Customizing the SAST settings +### Customizing the SAST settings The SAST settings can be changed through environment variables by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. @@ -134,7 +125,7 @@ variables: Because the template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline configuration, the last mention of the variable will take precedence. -#### Overriding the SAST template +### Overriding the SAST template If you want to override the job definition (for example, change properties like `variables` or `dependencies`), you need to declare a `sast` job after the @@ -149,77 +140,61 @@ sast: CI_DEBUG_TRACE: "true" ``` -### Manual job definition for GitLab 11.5 and later +### Available variables -For GitLab 11.5 and GitLab Runner 11.5 and later, the following `sast` -job can be added: +SAST can be [configured](#customizing-the-sast-settings) using environment variables. -```yaml -sast: - stage: test - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - allow_failure: true - services: - - docker:stable-dind - script: - - export SAST_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} - - | - docker run \ - --env SAST_ANALYZER_IMAGES \ - --env SAST_ANALYZER_IMAGE_PREFIX \ - --env SAST_ANALYZER_IMAGE_TAG \ - --env SAST_DEFAULT_ANALYZERS \ - --env SAST_EXCLUDED_PATHS \ - --env SAST_BANDIT_EXCLUDED_PATHS \ - --env SAST_BRAKEMAN_LEVEL \ - --env SAST_GOSEC_LEVEL \ - --env SAST_FLAWFINDER_LEVEL \ - --env SAST_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ - --env SAST_PULL_ANALYZER_IMAGE_TIMEOUT \ - --env SAST_RUN_ANALYZER_TIMEOUT \ - --volume "$PWD:/code" \ - --volume /var/run/docker.sock:/var/run/docker.sock \ - "registry.gitlab.com/gitlab-org/security-products/sast:$SAST_VERSION" /app/bin/run /code - dependencies: [] - artifacts: - reports: - sast: gl-sast-report.json -``` +#### Docker images -You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/sast#settings) -via `docker run --env` to customize your job execution. +The following are Docker image-related variables. -### Manual job definition for GitLab 11.4 and earlier (deprecated) +| Environment variable | Description | +|-------------------------------|--------------------------------------------------------------------------------| +| `SAST_ANALYZER_IMAGES` | Comma separated list of custom images. Default images are still enabled. Read more about [customizing analyzers](analyzers.md). | +| `SAST_ANALYZER_IMAGE_PREFIX` | Override the name of the Docker registry providing the default images (proxy). Read more about [customizing analyzers](analyzers.md). | +| `SAST_ANALYZER_IMAGE_TAG` | Override the Docker tag of the default images. Read more about [customizing analyzers](analyzers.md). | +| `SAST_DEFAULT_ANALYZERS` | Override the names of default images. Read more about [customizing analyzers](analyzers.md). | +| `SAST_PULL_ANALYZER_IMAGES` | Pull the images from the Docker registry (set to 0 to disable). Read more about [customizing analyzers](analyzers.md). | -CAUTION: **Deprecated:** -Before GitLab 11.5, the SAST job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained, they have been deprecated -and may be removed in the next major release, GitLab 12.0. You are strongly -advised to update your current `.gitlab-ci.yml` configuration to reflect that change. +### Vulnerability filters -For GitLab 11.4 and earlier, the SAST job should look like: +Some analyzers make it possible to filter out vulnerabilities under a given threshold. -```yaml -sast: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - allow_failure: true - services: - - docker:stable-dind - script: - - export SAST_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} - - docker run - --env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}" - --volume "$PWD:/code" - --volume /var/run/docker.sock:/var/run/docker.sock - "registry.gitlab.com/gitlab-org/security-products/sast:$SAST_VERSION" /app/bin/run /code - artifacts: - paths: [gl-sast-report.json] -``` +| Environment variable | Default value | Description | Example usage | +|----------------------|---------------|-------------|---| +| `SAST_BANDIT_EXCLUDED_PATHS` | - | comma-separated list of paths to exclude from scan. Uses Python's [`fnmatch` syntax](https://docs.python.org/2/library/fnmatch.html) | | +| `SAST_BRAKEMAN_LEVEL` | 1 | Ignore Brakeman vulnerabilities under given confidence level. Integer, 1=Low 3=High. | | +| `SAST_FLAWFINDER_LEVEL` | 1 | Ignore Flawfinder vulnerabilities under given risk level. Integer, 0=No risk, 5=High risk. | | +| `SAST_GITLEAKS_ENTROPY_LEVEL` | 8.0 | Minimum entropy for secret detection. Float, 0.0 = low, 8.0 = high. | | +| `SAST_GOSEC_LEVEL` | 0 | Ignore gosec vulnerabilities under given confidence level. Integer, 0=Undefined, 1=Low, 2=Medium, 3=High. | | +| `SAST_EXCLUDED_PATHS` | - | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs, file or folder paths. Parent directories will also match patterns. | `SAST_EXCLUDED_PATHS=doc,spec` | + +### Timeouts + +The following variables configure timeouts. + +| Environment variable | Default value | Description | +|----------------------|---------------|-------------| +| `SAST_DOCKER_CLIENT_NEGOTIATION_TIMEOUT` | 2m | Time limit for Docker client negotiation. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". For example, "300ms", "1.5h" or "2h45m". | +| `SAST_PULL_ANALYZER_IMAGE_TIMEOUT` | 5m | Time limit when pulling the image of an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". For example, "300ms", "1.5h" or "2h45m". | +| `SAST_RUN_ANALYZER_TIMEOUT` | 20m | Time limit when running an analyzer. Timeouts are parsed using Go's [`ParseDuration`](https://golang.org/pkg/time/#ParseDuration). Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". For example, "300ms", "1.5h" or "2h45m".| + +### Analyzer settings + +Some analyzers can be customized with environment variables. + +| Environment variable | Analyzer | Description | +|-------------------------|----------|----------| +| `ANT_HOME` | spotbugs | The `ANT_HOME` environment variable. | +| `ANT_PATH` | spotbugs | Path to the `ant` executable. | +| `GRADLE_PATH` | spotbugs | Path to the `gradle` executable. | +| `JAVA_OPTS` | spotbugs | Additional arguments for the `java` executable. | +| `JAVA_PATH` | spotbugs | Path to the `java` executable. | +| `MAVEN_CLI_OPTS` | spotbugs | Additional arguments for the `mvn` or `mvnw` executable. | +| `MAVEN_PATH` | spotbugs | Path to the `mvn` executable. | +| `MAVEN_REPO_PATH` | spotbugs | Path to the Maven local repository (shortcut for the `maven.repo.local` property). | +| `SBT_PATH` | spotbugs | Path to the `sbt` executable. | +| `FAIL_NEVER` | spotbugs | Set to `1` to ignore compilation failure. | ## Reports JSON format @@ -368,3 +343,15 @@ Once a vulnerability is found, you can interact with it. Read more on how to For more information about the vulnerabilities database update, check the [maintenance table](../index.md#maintenance-and-update-of-the-vulnerabilities-database). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/application_security/security_dashboard/img/dashboard.png b/doc/user/application_security/security_dashboard/img/dashboard.png Binary files differdeleted file mode 100644 index a75168b1ce4..00000000000 --- a/doc/user/application_security/security_dashboard/img/dashboard.png +++ /dev/null diff --git a/doc/user/application_security/security_dashboard/img/group_security_dashboard.png b/doc/user/application_security/security_dashboard/img/group_security_dashboard.png Binary files differnew file mode 100644 index 00000000000..85ab124f74c --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/group_security_dashboard.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard.png Binary files differindex f0dad6c54d0..baa136fd885 100644 --- a/doc/user/application_security/security_dashboard/img/project_security_dashboard.png +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 3b01fe66e03..ac8c1ac0354 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # GitLab Security Dashboard **(ULTIMATE)** The Security Dashboard is a good place to get an overview of all the security @@ -16,9 +20,9 @@ To benefit from the Security Dashboard you must first configure one of the The Security Dashboard supports the following reports: - [Container Scanning](../container_scanning/index.md) -- [DAST](../dast/index.md) +- [Dynamic Application Security Testing](../dast/index.md) - [Dependency Scanning](../dependency_scanning/index.md) -- [SAST](../sast/index.md) +- [Static Application Security Testing](../sast/index.md) ## Requirements @@ -26,8 +30,8 @@ To use the project or group security dashboard: 1. At least one project inside a group must be configured with at least one of the [supported reports](#supported-reports). -2. The configured jobs must use the [new `reports` syntax](../../../ci/yaml/README.md#artifactsreports). -3. [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or newer must be used. +1. The configured jobs must use the [new `reports` syntax](../../../ci/yaml/README.md#artifactsreports). +1. [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or newer must be used. If you're using the shared Runners on GitLab.com, this is already the case. ## Project Security Dashboard @@ -43,13 +47,13 @@ for your project. Use it to find and fix vulnerabilities affecting the ## Group Security Dashboard > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6709) in - [GitLab Ultimate](https://about.gitlab.com/pricing) 11.5. +> [GitLab Ultimate](https://about.gitlab.com/pricing) 11.5. The group Security Dashboard gives an overview of the vulnerabilities of all the projects in a group and its subgroups. First, navigate to the Security Dashboard found under your group's -**Overview > Security Dashboard**. +**Security** tab. Once you're on the dashboard, at the top you should see a series of filters for: @@ -58,7 +62,7 @@ Once you're on the dashboard, at the top you should see a series of filters for: - Report type - Project - + Selecting one or more filters will filter the results in this page. The first section is an overview of all the vulnerabilities, grouped by severity. @@ -102,3 +106,15 @@ That way, reports are created even if no code change happens. When using [Auto DevOps](../../../topics/autodevops/index.md), use [special environment variables](../../../topics/autodevops/index.md#environment-variables) to configure daily security scans. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index 06cc64b209d..862316b57da 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -170,6 +170,7 @@ Attach a block or paragraph to a list item using a list continuation (which you * [x] checked * [ ] not checked ``` + #### Callout ```asciidoc @@ -188,6 +189,7 @@ first term:: description of first term second term:: description of second term ``` + ### Document Structure #### Header @@ -197,6 +199,7 @@ description of second term Author Name <author@example.org> v1.0, 2019-01-01 ``` + #### Sections ```asciidoc @@ -217,6 +220,7 @@ include::basics.adoc[] // define -a allow-uri-read to allow content to be read from URI include::https://example.org/installation.adoc[] ``` + ### Blocks ```asciidoc @@ -273,11 +277,11 @@ source - a listing that is embellished with (colorized) syntax highlighting ---- ``` -```asciidoc +~~~asciidoc \```language fenced code - a shorthand syntax for the source block \``` -``` +~~~ ```asciidoc [,attribution,citetitle] @@ -369,4 +373,3 @@ video::300817511[vimeo] // page break <<< ``` - diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md index e4fd08a582c..1b1d126b726 100644 --- a/doc/user/award_emojis.md +++ b/doc/user/award_emojis.md @@ -7,7 +7,7 @@ When you're collaborating online, you get fewer opportunities for high-fives and thumbs-ups. Emoji can be awarded to [issues](project/issues/index.md), [merge requests](project/merge_requests/index.md), -[snippets](snippets.md), and anywhere you can have a discussion. +[snippets](snippets.md), and anywhere you can have a thread.  diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 2246ea8ed5a..096730f800c 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -1,15 +1,18 @@ # GitLab Managed Apps GitLab provides **GitLab Managed Apps**, a one-click install for various applications which can -be added directly to your configured cluster. These applications are -needed for [Review Apps](../../ci/review_apps/index.md) and -[deployments](../../ci/environments.md) when using [Auto DevOps](../../topics/autodevops/index.md). +be added directly to your configured cluster. + +These applications are needed for [Review Apps](../../ci/review_apps/index.md) +and [deployments](../../ci/environments.md) when using [Auto DevOps](../../topics/autodevops/index.md). + You can install them after you -[create a cluster](../project/clusters/index.md#adding-and-creating-a-new-gke-cluster-via-gitlab). +[create a cluster](../project/clusters/index.md#adding-and-removing-clusters). ## Installing applications Applications managed by GitLab will be installed onto the `gitlab-managed-apps` namespace. + This namespace: - Is different from the namespace used for project deployments. @@ -19,8 +22,10 @@ This namespace: To see a list of available applications to install: 1. For a: - - Project-level cluster, navigate to your project's **Operations > Kubernetes**. - - Group-level cluster, navigate to your group's **Kubernetes** page. + - [Project-level cluster](../project/clusters/index.md), + navigate to your project's **Operations > Kubernetes**. + - [Group-level cluster](../group/clusters/index.md), + navigate to your group's **Kubernetes** page. Install Helm first as it's used to install other applications. @@ -94,7 +99,7 @@ CI/CD](../../ci/README.md), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security -implications](../project/clusters/index.md/#security-implications) before doing so. +implications](../project/clusters/index.md#security-implications) before doing so. NOTE: **Note:** The @@ -159,9 +164,9 @@ file. When installing JupyterHub onto your Kubernetes cluster, [JupyterLab's Git extension](https://github.com/jupyterlab/jupyterlab-git) is automatically provisioned and configured using the authenticated user's: -- Name -- Email -- Newly created access token +- Name. +- Email. +- Newly created access token. JupyterLab's Git extension enables full version control of your notebooks as well as issuance of Git commands within Jupyter. Git commands can be issued via the **Git** tab on the left panel or via Jupyter's command line prompt. @@ -232,8 +237,10 @@ The applications below can be upgraded. To upgrade an application: 1. For a: - - Project-level cluster, navigate to your project's **Operations > Kubernetes**. - - Group-level cluster, navigate to your group's **Kubernetes** page. + - [Project-level cluster](../project/clusters/index.md), + navigate to your project's **Operations > Kubernetes**. + - [Group-level cluster](../group/clusters/index.md), + navigate to your group's **Kubernetes** page. 1. Select your cluster. 1. If an upgrade is available, the **Upgrade** button is displayed. Click the button to upgrade. @@ -251,15 +258,21 @@ The applications below can be uninstalled. | Application | GitLab version | Notes | | ----------- | -------------- | ----- | +| Cert-Manager | 12.2+ | The associated private key will be deleted and cannot be restored. Deployed applications will continue to use HTTPS, but certificates will not be renewed. Before uninstalling, you may wish to [back up your configuration](https://docs.cert-manager.io/en/latest/tasks/backup-restore-crds.html) or [revoke your certificates](https://letsencrypt.org/docs/revoking/) | +| GitLab Runner | 12.2+ | Any running pipelines will be canceled. | +| Helm | 12.2+ | The associated Tiller pod will be deleted and cannot be restored. | | Ingress | 12.1+ | The associated load balancer and IP will be deleted and cannot be restored. Furthermore, it can only be uninstalled if JupyterHub is not installed. | | JupyterHub | 12.1+ | All data not committed to GitLab will be deleted and cannot be restored. | +| Knative | 12.1+ | The associated IP will be deleted and cannot be restored. | | Prometheus | 11.11+ | All data will be deleted and cannot be restored. | To uninstall an application: 1. For a: - - Project-level cluster, navigate to your project's **Operations > Kubernetes**. - - Group-level cluster, navigate to your group's **Kubernetes** page. + - [Project-level cluster](../project/clusters/index.md), + navigate to your project's **Operations > Kubernetes**. + - [Group-level cluster](../group/clusters/index.md), + navigate to your group's **Kubernetes** page. 1. Select your cluster. 1. Click the **Uninstall** button for the application. diff --git a/doc/user/clusters/img/jupyter-gitclone.png b/doc/user/clusters/img/jupyter-gitclone.png Binary files differindex 41d467f806a..aff194dea43 100644 --- a/doc/user/clusters/img/jupyter-gitclone.png +++ b/doc/user/clusters/img/jupyter-gitclone.png diff --git a/doc/user/discussions/img/automatically_resolve_outdated_discussions.png b/doc/user/discussions/img/automatically_resolve_outdated_discussions.png Binary files differindex ba129e7a618..a6fc4b0aef1 100644 --- a/doc/user/discussions/img/automatically_resolve_outdated_discussions.png +++ b/doc/user/discussions/img/automatically_resolve_outdated_discussions.png diff --git a/doc/user/discussions/img/btn_new_issue_for_all_discussions.png b/doc/user/discussions/img/btn_new_issue_for_all_discussions.png Binary files differdeleted file mode 100644 index 3306bf2e60e..00000000000 --- a/doc/user/discussions/img/btn_new_issue_for_all_discussions.png +++ /dev/null diff --git a/doc/user/discussions/img/btn_new_issue_for_all_threads.png b/doc/user/discussions/img/btn_new_issue_for_all_threads.png Binary files differnew file mode 100644 index 00000000000..5d86e0ca016 --- /dev/null +++ b/doc/user/discussions/img/btn_new_issue_for_all_threads.png diff --git a/doc/user/discussions/img/commit_comment_mr_context.png b/doc/user/discussions/img/commit_comment_mr_context.png Binary files differindex b363e0035e8..68f58a57757 100644 --- a/doc/user/discussions/img/commit_comment_mr_context.png +++ b/doc/user/discussions/img/commit_comment_mr_context.png diff --git a/doc/user/discussions/img/commit_comment_mr_discussions_tab.png b/doc/user/discussions/img/commit_comment_mr_discussions_tab.png Binary files differindex 2b06cdcc055..d88f08eae26 100644 --- a/doc/user/discussions/img/commit_comment_mr_discussions_tab.png +++ b/doc/user/discussions/img/commit_comment_mr_discussions_tab.png diff --git a/doc/user/discussions/img/discussion_comment.png b/doc/user/discussions/img/discussion_comment.png Binary files differindex 206ddebf54b..3fec5962363 100644 --- a/doc/user/discussions/img/discussion_comment.png +++ b/doc/user/discussions/img/discussion_comment.png diff --git a/doc/user/discussions/img/discussion_view.png b/doc/user/discussions/img/discussion_view.png Binary files differdeleted file mode 100644 index 3a2b766ed7e..00000000000 --- a/doc/user/discussions/img/discussion_view.png +++ /dev/null diff --git a/doc/user/discussions/img/image_resolved_discussion.png b/doc/user/discussions/img/image_resolved_discussion.png Binary files differindex ed00b5c77fe..f6e5a3b66ae 100644 --- a/doc/user/discussions/img/image_resolved_discussion.png +++ b/doc/user/discussions/img/image_resolved_discussion.png diff --git a/doc/user/discussions/img/make_suggestion.png b/doc/user/discussions/img/make_suggestion.png Binary files differindex 20acc1417da..a24e29770aa 100644 --- a/doc/user/discussions/img/make_suggestion.png +++ b/doc/user/discussions/img/make_suggestion.png diff --git a/doc/user/discussions/img/merge_request_commits_tab.png b/doc/user/discussions/img/merge_request_commits_tab.png Binary files differindex 41a3648f390..267f3a720dc 100644 --- a/doc/user/discussions/img/merge_request_commits_tab.png +++ b/doc/user/discussions/img/merge_request_commits_tab.png diff --git a/doc/user/discussions/img/mr_review_resolve.png b/doc/user/discussions/img/mr_review_resolve.png Binary files differindex 34176f3fa8e..ced33682459 100644 --- a/doc/user/discussions/img/mr_review_resolve.png +++ b/doc/user/discussions/img/mr_review_resolve.png diff --git a/doc/user/discussions/img/mr_review_resolve2.png b/doc/user/discussions/img/mr_review_resolve2.png Binary files differindex e4adb5f2c2d..2f0be3b6d06 100644 --- a/doc/user/discussions/img/mr_review_resolve2.png +++ b/doc/user/discussions/img/mr_review_resolve2.png diff --git a/doc/user/discussions/img/mr_review_second_comment.png b/doc/user/discussions/img/mr_review_second_comment.png Binary files differindex 920ea05ad66..f345c52e941 100644 --- a/doc/user/discussions/img/mr_review_second_comment.png +++ b/doc/user/discussions/img/mr_review_second_comment.png diff --git a/doc/user/discussions/img/mr_review_second_comment_added.png b/doc/user/discussions/img/mr_review_second_comment_added.png Binary files differindex 1fb54348547..61b45431c9e 100644 --- a/doc/user/discussions/img/mr_review_second_comment_added.png +++ b/doc/user/discussions/img/mr_review_second_comment_added.png diff --git a/doc/user/discussions/img/mr_review_start.png b/doc/user/discussions/img/mr_review_start.png Binary files differindex 38b44bda0d2..08b4c6bb82b 100644 --- a/doc/user/discussions/img/mr_review_start.png +++ b/doc/user/discussions/img/mr_review_start.png diff --git a/doc/user/discussions/img/mr_review_unresolve.png b/doc/user/discussions/img/mr_review_unresolve.png Binary files differindex da895ceb89f..4bef38f7808 100644 --- a/doc/user/discussions/img/mr_review_unresolve.png +++ b/doc/user/discussions/img/mr_review_unresolve.png diff --git a/doc/user/discussions/img/mr_review_unresolve2.png b/doc/user/discussions/img/mr_review_unresolve2.png Binary files differindex a824b806e4a..79da61bb556 100644 --- a/doc/user/discussions/img/mr_review_unresolve2.png +++ b/doc/user/discussions/img/mr_review_unresolve2.png diff --git a/doc/user/discussions/img/multi-line-suggestion-preview.png b/doc/user/discussions/img/multi-line-suggestion-preview.png Binary files differindex 4288d0ba034..476c50b9098 100644 --- a/doc/user/discussions/img/multi-line-suggestion-preview.png +++ b/doc/user/discussions/img/multi-line-suggestion-preview.png diff --git a/doc/user/discussions/img/multi-line-suggestion-syntax.png b/doc/user/discussions/img/multi-line-suggestion-syntax.png Binary files differindex df0c99b84ef..80424d1f056 100644 --- a/doc/user/discussions/img/multi-line-suggestion-syntax.png +++ b/doc/user/discussions/img/multi-line-suggestion-syntax.png diff --git a/doc/user/discussions/img/new_issue_for_thread.png b/doc/user/discussions/img/new_issue_for_thread.png Binary files differnew file mode 100644 index 00000000000..28b76adf7fe --- /dev/null +++ b/doc/user/discussions/img/new_issue_for_thread.png diff --git a/doc/user/discussions/img/onion_skin_view.png b/doc/user/discussions/img/onion_skin_view.png Binary files differindex 91c3b396844..81bb4a2c85a 100644 --- a/doc/user/discussions/img/onion_skin_view.png +++ b/doc/user/discussions/img/onion_skin_view.png diff --git a/doc/user/discussions/img/only_allow_merge_if_all_threads_are_resolved.png b/doc/user/discussions/img/only_allow_merge_if_all_threads_are_resolved.png Binary files differnew file mode 100644 index 00000000000..bd0aaca79b2 --- /dev/null +++ b/doc/user/discussions/img/only_allow_merge_if_all_threads_are_resolved.png diff --git a/doc/user/discussions/img/pending_review_comment.png b/doc/user/discussions/img/pending_review_comment.png Binary files differindex 916ef5b7452..70a66b3f4f0 100644 --- a/doc/user/discussions/img/pending_review_comment.png +++ b/doc/user/discussions/img/pending_review_comment.png diff --git a/doc/user/discussions/img/preview_issue_for_thread.png b/doc/user/discussions/img/preview_issue_for_thread.png Binary files differnew file mode 100644 index 00000000000..a9d7ab598be --- /dev/null +++ b/doc/user/discussions/img/preview_issue_for_thread.png diff --git a/doc/user/discussions/img/preview_issue_for_threads.png b/doc/user/discussions/img/preview_issue_for_threads.png Binary files differnew file mode 100644 index 00000000000..8757decb178 --- /dev/null +++ b/doc/user/discussions/img/preview_issue_for_threads.png diff --git a/doc/user/discussions/img/reply_to_comment_button.png b/doc/user/discussions/img/reply_to_comment_button.png Binary files differindex d362b19785c..d327d1c3e27 100644 --- a/doc/user/discussions/img/reply_to_comment_button.png +++ b/doc/user/discussions/img/reply_to_comment_button.png diff --git a/doc/user/discussions/img/resolve_comment_button.png b/doc/user/discussions/img/resolve_comment_button.png Binary files differindex 7c19fac31a2..0a3ed03a69c 100644 --- a/doc/user/discussions/img/resolve_comment_button.png +++ b/doc/user/discussions/img/resolve_comment_button.png diff --git a/doc/user/discussions/img/resolve_thread_button.png b/doc/user/discussions/img/resolve_thread_button.png Binary files differnew file mode 100644 index 00000000000..ca0a3e50550 --- /dev/null +++ b/doc/user/discussions/img/resolve_thread_button.png diff --git a/doc/user/discussions/img/resolve_thread_issue_notice.png b/doc/user/discussions/img/resolve_thread_issue_notice.png Binary files differnew file mode 100644 index 00000000000..30a65b8fbd4 --- /dev/null +++ b/doc/user/discussions/img/resolve_thread_issue_notice.png diff --git a/doc/user/discussions/img/resolve_thread_open_issue.png b/doc/user/discussions/img/resolve_thread_open_issue.png Binary files differnew file mode 100644 index 00000000000..2dd4ea3cb1b --- /dev/null +++ b/doc/user/discussions/img/resolve_thread_open_issue.png diff --git a/doc/user/discussions/img/review_comment_quickactions.png b/doc/user/discussions/img/review_comment_quickactions.png Binary files differindex bd9880c329a..276def6381f 100644 --- a/doc/user/discussions/img/review_comment_quickactions.png +++ b/doc/user/discussions/img/review_comment_quickactions.png diff --git a/doc/user/discussions/img/review_preview.png b/doc/user/discussions/img/review_preview.png Binary files differindex 4bf53a81b9c..e69a58dbb91 100644 --- a/doc/user/discussions/img/review_preview.png +++ b/doc/user/discussions/img/review_preview.png diff --git a/doc/user/discussions/img/suggestion.png b/doc/user/discussions/img/suggestion.png Binary files differindex 68a67e6ae5e..f7962305a15 100644 --- a/doc/user/discussions/img/suggestion.png +++ b/doc/user/discussions/img/suggestion.png diff --git a/doc/user/discussions/img/swipe_view.png b/doc/user/discussions/img/swipe_view.png Binary files differindex 82d6e52173c..e6f5e5053af 100644 --- a/doc/user/discussions/img/swipe_view.png +++ b/doc/user/discussions/img/swipe_view.png diff --git a/doc/user/discussions/img/thread_view.png b/doc/user/discussions/img/thread_view.png Binary files differnew file mode 100644 index 00000000000..e2db44aa604 --- /dev/null +++ b/doc/user/discussions/img/thread_view.png diff --git a/doc/user/discussions/img/threads_resolved.png b/doc/user/discussions/img/threads_resolved.png Binary files differnew file mode 100644 index 00000000000..ffb1233f2ee --- /dev/null +++ b/doc/user/discussions/img/threads_resolved.png diff --git a/doc/user/discussions/img/two_up_view.png b/doc/user/discussions/img/two_up_view.png Binary files differindex d9e90708e87..3b6ddfbe1be 100644 --- a/doc/user/discussions/img/two_up_view.png +++ b/doc/user/discussions/img/two_up_view.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index c6bc580fb8f..6891682141c 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -1,4 +1,4 @@ -# Discussions +# Threads The ability to contribute conversationally is offered throughout GitLab. @@ -12,7 +12,7 @@ You can leave a comment in the following places: - commit diffs There are standard comments, and you also have the option to create a comment -in the form of a threaded discussion. A comment can also be [turned into a discussion](#start-a-discussion-by-replying-to-a-standard-comment) +in the form of a thread. A comment can also be [turned into a thread](#start-a-thread-by-replying-to-a-standard-comment) when it receives a reply. The comment area supports [Markdown] and [quick actions]. You can edit your own @@ -21,187 +21,190 @@ higher can also edit a comment made by someone else. You can also reply to a comment notification email to reply to the comment if [Reply by email] is configured for your GitLab instance. Replying to a standard comment -creates another standard comment. Replying to a discussion comment creates a reply in the -discussion thread. Email replies support [Markdown] and [quick actions], just as if you replied from the web. +creates another standard comment. Replying to a threaded comment creates a reply in the thread. Email replies support +[Markdown] and [quick actions], just as if you replied from the web. -## Resolvable comments and discussions +## Resolvable comments and threads > **Notes:** > > - The main feature was [introduced][ce-5022] in GitLab 8.11. -> - Resolvable discussions can be added only to merge request diffs. +> - Resolvable threads can be added only to merge request diffs. -Discussion resolution helps keep track of progress during planning or code review. +Thread resolution helps keep track of progress during planning or code review. -Every standard comment or discussion thread in merge requests, commits, commit diffs, and +Every standard comment or thread in merge requests, commits, commit diffs, and snippets is initially displayed as unresolved. They can then be individually resolved by anyone with at least Developer access to the project or by the author of the change being reviewed. -The need to resolve all standard comments or discussions prevents you from forgetting -to address feedback and lets you hide discussions that are no longer relevant. +The need to resolve all standard comments or threads prevents you from forgetting +to address feedback and lets you hide threads that are no longer relevant. -!["A discussion between two people on a piece of code"][discussion-view] + -### Commit discussions in the context of a merge request +### Commit threads in the context of a merge request > [Introduced][ce-31847] in GitLab 10.3. -For reviewers with commit-based workflow, it may be useful to add discussions to -specific commit diffs in the context of a merge request. These discussions will +For reviewers with commit-based workflow, it may be useful to add threads to +specific commit diffs in the context of a merge request. These threads will persist through a commit ID change when: - force-pushing after a rebase - amending a commit -This functionality is also demonstrated in the video [How to use Merge Request Commit Discussions](https://www.youtube.com/watch?v=TviJH6oRboo). - -To create a commit diff discussion: +To create a commit diff thread: 1. Navigate to the merge request **Commits** tab. A list of commits that constitute the merge request will be shown. -  +  1. Navigate to a specific commit, click on the **Changes** tab (where you will only be presented diffs from the selected commit), and leave a comment. -  +  -1. Any discussions created this way will be shown in the merge request's +1. Any threads created this way will be shown in the merge request's **Discussions** tab and are resolvable. -  +  -Discussions created this way will only appear in the original merge request +Threads created this way will only appear in the original merge request and not when navigating to that commit under your project's **Repository > Commits** page. TIP: **Tip:** -When a link of a commit reference is found in a discussion inside a merge +When a link of a commit reference is found in a thread inside a merge request, it will be automatically converted to a link in the context of the current merge request. -### Jumping between unresolved discussions +### Jumping between unresolved threads When a merge request has a large number of comments it can be difficult to track -what remains unresolved. You can jump between unresolved discussions with the -Jump button next to the Reply field on a discussion. +what remains unresolved. You can jump between unresolved threads with the +Jump button next to the Reply field on a thread. + +You can also jump to the first unresolved thread from the button next to the +resolved threads tracker. -You can also jump to the first unresolved discussion from the button next to the -resolved discussions tracker. +You can also use keyboard shortcuts to navigate among threads: -!["3/4 discussions resolved"][discussions-resolved] +- Use <kbd>n</kbd> to jump to the next unresolved thread. +- Use <kbd>p</kbd> to jump to the previous unresolved thread. -### Marking a comment or discussion as resolved + -You can mark a discussion as resolved by clicking the **Resolve discussion** -button at the bottom of the discussion. +### Marking a comment or thread as resolved -!["Resolve discussion" button][resolve-discussion-button] +You can mark a thread as resolved by clicking the **Resolve thread** +button at the bottom of the thread. + + Alternatively, you can mark each comment as resolved individually. -!["Resolve comment" button][resolve-comment-button] + -### Move all unresolved discussions in a merge request to an issue +### Move all unresolved threads in a merge request to an issue > [Introduced][ce-8266] in GitLab 9.1 -To continue all open discussions from a merge request in a new issue, click the -**Resolve all discussions in new issue** button. +To continue all open threads from a merge request in a new issue, click the +**Resolve all threads in new issue** button. - + -Alternatively, when your project only accepts merge requests [when all discussions -are resolved](#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved), +Alternatively, when your project only accepts merge requests [when all threads +are resolved](#only-allow-merge-requests-to-be-merged-if-all-threads-are-resolved), there will be an **open an issue to resolve them later** link in the merge request widget. - + This will prepare an issue with its content referring to the merge request and -the unresolved discussions. +the unresolved threads. - + -Hitting **Submit issue** will cause all discussions to be marked as resolved and +Hitting **Submit issue** will cause all threads to be marked as resolved and add a note referring to the newly created issue. - + You can now proceed to merge the merge request from the UI. -### Moving a single discussion to a new issue +### Moving a single thread to a new issue > [Introduced][ce-8266] in GitLab 9.1 -To create a new issue for a single discussion, you can use the **Resolve this -discussion in a new issue** button. +To create a new issue for a single thread, you can use the **Resolve this +thread in a new issue** button. - + This will direct you to a new issue prefilled with the content of the -discussion, similar to the issues created for delegating multiple -discussions at once. Saving the issue will mark the discussion as resolved and -add a note to the merge request discussion referencing the new issue. +thread, similar to the issues created for delegating multiple +threads at once. Saving the issue will mark the thread as resolved and +add a note to the merge request thread referencing the new issue. - + -### Only allow merge requests to be merged if all discussions are resolved +### Only allow merge requests to be merged if all threads are resolved > [Introduced][ce-7125] in GitLab 8.14. -You can prevent merge requests from being merged until all discussions are +You can prevent merge requests from being merged until all threads are resolved. Navigate to your project's settings page, select the -**Only allow merge requests to be merged if all discussions are resolved** check +**Only allow merge requests to be merged if all threads are resolved** check box and hit **Save** for the changes to take effect. - + -From now on, you will not be able to merge from the UI until all discussions +From now on, you will not be able to merge from the UI until all threads are resolved. - + -### Automatically resolve merge request diff discussions when they become outdated +### Automatically resolve merge request diff threads when they become outdated > [Introduced][ce-14053] in GitLab 10.0. -You can automatically resolve merge request diff discussions on lines modified +You can automatically resolve merge request diff threads on lines modified with a new push. Navigate to your project's settings page, select the **Automatically resolve -merge request diffs discussions on lines changed with a push** check box and hit +merge request diffs threads on lines changed with a push** check box and hit **Save** for the changes to take effect. - + -From now on, any discussions on a diff will be resolved by default if a push -makes that diff section outdated. Discussions on lines that don't change and -top-level resolvable discussions are not automatically resolved. +From now on, any threads on a diff will be resolved by default if a push +makes that diff section outdated. Threads on lines that don't change and +top-level resolvable threads are not automatically resolved. -## Commit discussions +## Commit threads -You can add comments and discussion threads to a particular commit under your +You can add comments and threads to a particular commit under your project's **Repository > Commits**. CAUTION: **Attention:** -Discussions created this way will be lost if the commit ID changes after a +Threads created this way will be lost if the commit ID changes after a force push. ## Threaded discussions > [Introduced][ce-7527] in GitLab 9.1. -While resolvable discussions are only available to merge request diffs, -discussions can also be added without a diff. You can start a specific -discussion which will look like a thread, on issues, commits, snippets, and +While resolvable threads are only available to merge request diffs, +threads can also be added without a diff. You can start a specific +thread which will look like a thread, on issues, commits, snippets, and merge requests. To start a threaded discussion, click on the **Comment** button toggle dropdown, -select **Start discussion** and click **Start discussion** when you're ready to +select **Start thread** and click **Start thread** when you're ready to post the comment.  @@ -209,56 +212,57 @@ post the comment. This will post a comment with a single thread to allow you to discuss specific comments in greater detail. - + -## Image discussions +## Image threads > [Introduced][ce-14061] in GitLab 10.1. -Sometimes a discussion is revolved around an image. With image discussions, -you can easily target a specific coordinate of an image and start a discussion -around it. Image discussions are available in merge requests and commit detail views. +Sometimes a thread is revolved around an image. With image threads, +you can easily target a specific coordinate of an image and start a thread +around it. Image threads are available in merge requests and commit detail views. -To start an image discussion, hover your mouse over the image. Your mouse pointer +To start an image thread, hover your mouse over the image. Your mouse pointer should convert into an icon, indicating that the image is available for commenting. -Simply click anywhere on the image to create a new discussion. +Simply click anywhere on the image to create a new thread. - + After you click on the image, a comment form will be displayed that would be the start -of your discussion. Once you save your comment, you will see a new badge displayed on -top of your image. This badge represents your discussion. +of your thread. Once you save your comment, you will see a new badge displayed on +top of your image. This badge represents your thread. >**Note:** -This discussion badge is typically associated with a number that is only used as a visual -reference for each discussion. In the merge request discussion tab, -this badge will be indicated with a comment icon since each discussion will render a new +This thread badge is typically associated with a number that is only used as a visual +reference for each thread. In the merge request thread tab, +this badge will be indicated with a comment icon since each thread will render a new image section. -Image discussions also work on diffs that replace an existing image. In this diff view -mode, you can toggle the different view modes and still see the discussion point badges. +Image threads also work on diffs that replace an existing image. In this diff view +mode, you can toggle the different view modes and still see the thread point badges. | 2-up | Swipe | Onion Skin | | :-----------: | :----------: | :----------: | |  |  |  | -Image discussions also work well with resolvable discussions. Resolved discussions +Image threads also work well with resolvable threads. Resolved threads on diffs (not on the merge request discussion tab) will appear collapsed on page load and will have a corresponding badge counter to match the counter on the image. - + ## Lock discussions > [Introduced][ce-14531] in GitLab 10.1. -For large projects with many contributors, it may be useful to stop discussions +For large projects with many contributors, it may be useful to stop threads in issues or merge requests in these scenarios: -- The project maintainer has already resolved the discussion and it is not helpful - for continued feedback. The project maintainer has already directed new conversation +- The project maintainer has already resolved the thread and it is not helpful + for continued feedback. +- The project maintainer has already directed new conversation to newer issues or merge requests. -- The people participating in the discussion are trolling, abusive, or otherwise +- The people participating in the thread are trolling, abusive, or otherwise being unproductive. In these cases, a user with Developer permissions or higher in the project can lock (and unlock) @@ -300,7 +304,7 @@ in an MR and click on the **Start a review** button. Once a review is started, you will see any comments that are part of this review marked `Pending`. All comments that are part of a review show two buttons: -- **Submit review**: Submits all comments that are part of the review, making them visible to other users. +- **Finish review**: Submits all comments that are part of the review, making them visible to other users. - **Add comment now**: Submits the specific comment as a regular comment instead of as part of the review.  @@ -317,20 +321,20 @@ This will add the comment to the review.  -### Resolving/Unresolving discussions +### Resolving/Unresolving threads -Review comments can also resolve/unresolve [resolvable discussions](#resolvable-comments-and-discussions). +Review comments can also resolve/unresolve [resolvable threads](#resolvable-comments-and-threads). When replying to a comment, you will see a checkbox that you can click in order to resolve or unresolve -the discussion once published. +the thread once published.  - -If a particular pending comment will resolve or unresolve the discussion, this will be shown on the pending +If a particular pending comment will resolve or unresolve the thread, this will be shown on the pending comment itself.  - + + ### Submitting a review @@ -339,8 +343,8 @@ bottom of the screen with two buttons: - **Discard**: Discards all comments that have not been submitted. - **Finish review**: Opens a list of comments ready to be submitted for review. - Clicking **Submit review** will publish all comments. Any quick actions - submitted are performed at this time. + Clicking **Submit review** will publish all comments. Any quick actions + submitted are performed at this time. Alternatively, every pending comment has a button to finish the entire review. @@ -356,7 +360,7 @@ Replying to this email will, consequentially, create a new comment on the associ > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/26723) in GitLab 11.5. For issues with many comments like activity notes and user comments, sometimes -finding useful information can be hard. There is a way to filter comments from single notes and discussions for merge requests and issues. +finding useful information can be hard. There is a way to filter comments from single notes and threads for merge requests and issues. From a merge request's **Discussion** tab, or from an epic/issue overview, find the filter's dropdown menu on the right side of the page, from which you can choose one of the following options: @@ -376,7 +380,7 @@ from any device you're logged into. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/18008) in GitLab 11.6. As a reviewer, you're able to suggest code changes with a simple -markdown syntax in Merge Request Diff discussions. Then, the +markdown syntax in Merge Request Diff threads. Then, the Merge Request author (or other users with appropriate [permission](../permissions.md)) is able to apply these suggestions with a click, which will generate a commit in @@ -385,21 +389,21 @@ the Merge Request authored by the user that applied them. 1. Choose a line of code to be changed, add a new comment, then click on the **Insert suggestion** icon in the toolbar: -  +  1. In the comment, add your suggestion to the pre-populated code block: -  +  1. Click **Comment**. - The suggestions in the comment can be applied by the merge request author - directly from the merge request: + The suggestions in the comment can be applied by the merge request author + directly from the merge request: -  +  Once the author applies a suggestion, it will be marked with the **Applied** label, -the discussion will be automatically resolved, and GitLab will create a new commit +the thread will be automatically resolved, and GitLab will create a new commit with the message `Apply suggestion to <file-name>` and push the suggested change directly into the codebase in the merge request's branch. [Developer permission](../permissions.md) is required to do so. @@ -413,8 +417,8 @@ Custom commit messages will be introduced by > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53310) in GitLab 11.10. Reviewers can also suggest changes to multiple lines with a single suggestion -within Merge Request diff discussions by adjusting the range offsets. The -offsets are relative to the position of the diff discussion, and specify the +within Merge Request diff threads by adjusting the range offsets. The +offsets are relative to the position of the diff thread, and specify the range to be replaced by the suggestion when it is applied.  @@ -430,25 +434,26 @@ Suggestions covering multiple lines are limited to 100 lines _above_ and 100 lines _below_ the commented diff line, allowing up to 200 changed lines per suggestion. -## Start a discussion by replying to a standard comment +## Start a thread by replying to a standard comment > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30299) in GitLab 11.9 -To reply to a standard (non-discussion) comment, you can use the **Reply to comment** button. +To reply to a standard (non-thread) comment, you can use the **Reply to comment** button.  -The **Reply to comment** button is only displayed if you have permissions to reply to an existing discussion, or start a discussion from a standard comment. +The **Reply to comment** button is only displayed if you have permissions to reply to an existing thread, or start a thread from a standard comment. Clicking on the **Reply to comment** button will bring the reply area into focus and you can type your reply.  -Replying to a non-discussion comment will convert the non-discussion comment to a -threaded discussion once the reply is submitted. This conversion is considered an edit +Replying to a non-thread comment will convert the non-thread comment to a +thread once the reply is submitted. This conversion is considered an edit to the original comment, so a note about when it was last edited will appear underneath it. -This feature only exists for Issues, Merge requests, and Epics. Commits, Snippets and Merge request diff discussions are not supported yet. +This feature only exists for Issues, Merge requests, and Epics. Commits, Snippets and Merge request diff threads are +not supported yet. [ce-5022]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5022 [ce-7125]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7125 diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 7858c419e04..af37cc896ad 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -43,13 +43,15 @@ Host gitlab.com Below are the settings for [GitLab Pages]. -| Setting | GitLab.com | Default | -| ----------------------- | ---------------- | ------------- | -| Domain name | `gitlab.io` | - | -| IP address | `35.185.44.232` | - | -| Custom domains support | yes | no | -| TLS certificates support| yes | no | - +| Setting | GitLab.com | Default | +| --------------------------- | ---------------- | ------------- | +| Domain name | `gitlab.io` | - | +| IP address | `35.185.44.232` | - | +| Custom domains support | yes | no | +| TLS certificates support | yes | no | +| Maximum size (uncompressed) | 1G | 100M | + +NOTE: **Note:** The maximum size of your Pages site is regulated by the artifacts maximum size which is part of [GitLab CI/CD](#gitlab-cicd). @@ -59,7 +61,7 @@ Below are the current settings regarding [GitLab CI/CD](../../ci/README.md). | Setting | GitLab.com | Default | | ----------- | ----------------- | ------------- | -| Artifacts maximum size | 1G | 100M | +| Artifacts maximum size (uncompressed) | 1G | 100M | | Artifacts [expiry time](../../ci/yaml/README.md#artifactsexpire_in) | kept forever | deleted after 30 days unless otherwise specified | ## Repository size limit @@ -81,13 +83,12 @@ IP based firewall can be configured by looking up all ## Shared Runners -Shared Runners on GitLab.com run in [autoscale mode] and powered by -Google Cloud Platform. Autoscaling means reduced -waiting times to spin up CI/CD jobs, and isolated VMs for each project, -thus maximizing security. -They're free to use for public open source projects and limited to 2000 CI -minutes per month per group for private projects. Read about all -[GitLab.com plans](https://about.gitlab.com/pricing/). +Shared Runners on GitLab.com run in [autoscale mode] and powered by Google Cloud Platform. +Autoscaling means reduced waiting times to spin up CI/CD jobs, and isolated VMs for each project, +thus maximizing security. They're free to use for public open source projects and limited +to 2000 CI minutes per month per group for private projects. More minutes +[can be purchased](../../subscriptions/index.md#extra-shared-runners-pipeline-minutes), if +needed. Read about all [GitLab.com plans](https://about.gitlab.com/pricing/). All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default @@ -113,57 +114,6 @@ Below are the shared Runners settings. The full contents of our `config.toml` are: -**DigitalOcean** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - [runners.machine] - IdleCount = 20 - IdleTime = 1800 - OffPeakPeriods = ["* * * * * sat,sun *"] - OffPeakTimezone = "UTC" - OffPeakIdleCount = 5 - OffPeakIdleTime = 1800 - MaxBuilds = 1 - MachineName = "srm-%s" - MachineDriver = "digitalocean" - MachineOptions = [ - "digitalocean-image=X", - "digitalocean-ssh-user=core", - "digitalocean-region=nyc1", - "digitalocean-size=s-2vcpu-2gb", - "digitalocean-private-networking", - "digitalocean-tags=shared_runners,gitlab_com", - "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR", - "digitalocean-access-token=DIGITAL_OCEAN_ACCESS_TOKEN", - ] - [runners.cache] - Type = "s3" - BucketName = "runner" - Insecure = true - Shared = true - ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER" - AccessKey = "ACCESS_KEY" - SecretKey = "ACCESS_SECRET_KEY" -``` - **Google Cloud Platform** ```toml @@ -179,20 +129,25 @@ sentry_dsn = "X" token = "SHARED_RUNNER_TOKEN" executor = "docker+machine" environment = [ - "DOCKER_DRIVER=overlay2" + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" ] limit = X [runners.docker] image = "ruby:2.5" privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] [runners.machine] - IdleCount = 20 - IdleTime = 1800 + IdleCount = 50 + IdleTime = 3600 OffPeakPeriods = ["* * * * * sat,sun *"] OffPeakTimezone = "UTC" - OffPeakIdleCount = 5 - OffPeakIdleTime = 1800 - MaxBuilds = 1 + OffPeakIdleCount = 15 + OffPeakIdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. MachineName = "srm-%s" MachineDriver = "google" MachineOptions = [ @@ -203,17 +158,18 @@ sentry_dsn = "X" "google-tags=gitlab-com,srm", "google-use-internal-ip", "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/issues/3214#note_82892928 "google-machine-image=PROJECT/global/images/IMAGE", - "engine-registry-mirror=http://INTERNAL_IP_OF_OUR_REGISTRY_MIRROR" + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 ] [runners.cache] - Type = "s3" - BucketName = "runner" - Insecure = true + Type = "gcs" Shared = true - ServerAddress = "INTERNAL_IP_OF_OUR_CACHE_SERVER" - AccessKey = "ACCESS_KEY" - SecretKey = "ACCESS_SECRET_KEY" + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" ``` ## Sidekiq @@ -223,7 +179,7 @@ and the following environment variables: | Setting | GitLab.com | Default | |-------- |----------- |-------- | -| `SIDEKIQ_MEMORY_KILLER_MAX_RSS` | `1000000` | `1000000` | +| `SIDEKIQ_MEMORY_KILLER_MAX_RSS` | `1000000` | `2000000` | | `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL` | `SIGKILL` | - | | `SIDEKIQ_LOG_ARGUMENTS` | `1` | - | @@ -298,6 +254,82 @@ Web front-ends: - `memory_limit_min` = 1024MiB - `memory_limit_max` = 1280MiB +## GitLab.com-specific rate limits + +NOTE: **Note:** +See [Rate limits](../../security/rate_limits.md) for administrator +documentation. + +IP blocks usually happen when GitLab.com receives unusual traffic from a single +IP address that the system views as potentially malicious based on rate limit +settings. After the unusual traffic ceases, the IP address will be automatically +released depending on the type of block, as described below. + +If you receive a `403 Forbidden` error for all requests to GitLab.com, please +check for any automated processes that may be triggering a block. For +assistance, contact [GitLab Support](https://support.gitlab.com) +with details, such as the affected IP address. + +### HAProxy API throttle + +GitLab.com responds with HTTP status code `429` to API requests that exceed 10 +requests +per second per IP address. + +The following example headers are included for all API requests: + +``` +RateLimit-Limit: 600 +RateLimit-Observed: 6 +RateLimit-Remaining: 594 +RateLimit-Reset: 1563325137 +RateLimit-ResetTime: Wed, 17 Jul 2019 00:58:57 GMT +``` + +Source: + +- Search for `rate_limit_http_rate_per_minute` and `rate_limit_sessions_per_second` in [GitLab.com's current HAProxy settings](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/blob/master/attributes/default.rb). + +### Rack Attack initializer + +Details of rate limits enforced by [Rack Attack](../../security/rack_attack.md). + +#### Protected paths throttle + +GitLab.com responds with HTTP status code `429` to POST requests at protected +paths that exceed 10 requests per **minute** per IP address. + +See the source below for which paths are protected. This includes user creation, +user confirmation, user sign in, and password reset. + +This header is included in responses to blocked requests: + +``` +Retry-After: 60 +``` + +Source: + +- Search for `rate_limit_requests_per_period`, `rate_limit_period`, and `rack_attack_protected_paths` in [GitLab.com's current Rails app settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). + +#### Git and container registry failed authentication ban + +GitLab.com responds with HTTP status code 403 for 1 hour, if 30 failed +authentication requests were received in a 3-minute period from a single IP address. + +This applies only to Git requests and container registry (`/jwt/auth`) requests +(combined). + +This limit is reset by requests that authenticate successfully. For example, 29 +failed authentication requests followed by 1 successful request, followed by 29 +more failed authentication requests would not trigger a ban. + +No response headers are provided. + +### Admin Area settings + +GitLab.com does not currently use these settings. + ## GitLab.com at scale In addition to the GitLab Enterprise Edition Omnibus install, GitLab.com uses diff --git a/doc/user/group/bulk_editing/img/bulk-editing.png b/doc/user/group/bulk_editing/img/bulk-editing.png Binary files differnew file mode 100644 index 00000000000..ca1662a781b --- /dev/null +++ b/doc/user/group/bulk_editing/img/bulk-editing.png diff --git a/doc/user/group/bulk_editing/index.md b/doc/user/group/bulk_editing/index.md new file mode 100644 index 00000000000..ea48b0b9fef --- /dev/null +++ b/doc/user/group/bulk_editing/index.md @@ -0,0 +1,31 @@ +# Bulk editing issues, merge requests, and epics at the group level **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7249) for issues in [GitLab Premium](https://about.gitlab.com/pricing/) 12.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/12719) for merge requests in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7250) for epics in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. + +## Editing milestones and labels + +> **Notes:** +> +> - A permission level of `Reporter` or higher is required in order to manage issues. +> - A permission level of `Developer` or higher is required in order to manage merge requests. +> - A permission level of `Reporter` or higher is required in order to manage epics. + +By using the bulk editing feature: + +- Milestones can be updated simultaneously across multiple issues or merge requests. +- Labels can be updated simultaneously across multiple issues, merge requests, or epics. + + + +To bulk update group issues, merge requests, or epics: + +1. Navigate to the issues, merge requests, or epics list. +1. Click **Edit issues**, **Edit merge requests**, or **Edit epics**. + - This will open a sidebar on the right-hand side where editable fields + for milestones and labels will be displayed. + - Checkboxes will also appear beside each issue, merge request, or epic. +1. Check the checkbox beside each issue, merge request, or epic to be edited. +1. Select the desired new values from the sidebar. +1. Click **Update all**. diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 0dffc216f8e..d41f44f85cc 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -5,7 +5,6 @@ type: reference # Group-level Kubernetes clusters > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) in GitLab 11.6. -> Group Cluster integration is currently in [Beta](https://about.gitlab.com/handbook/product/#alpha-beta-ga). ## Overview @@ -87,7 +86,7 @@ The domain should have a wildcard DNS configured to the Ingress IP address. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the -[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium) +[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables) work. While evaluating which environment matches the environment scope of a @@ -138,6 +137,13 @@ The result will then be: - The Staging cluster will be used for the `deploy to staging` job. - The Production cluster will be used for the `deploy to production` job. +## Security of Runners + +For important information about securely configuring GitLab Runners, see +[Security of +Runners](../../project/clusters/index.md#security-of-gitlab-runners) +documentation for project-level clusters. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 2d37fc375db..a0dd5cd497d 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -4,7 +4,8 @@ type: reference # Contribution Analytics **(STARTER)** -> Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. +> - Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3090) for subgroups in GitLab 12.2. ## Overview diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 8a85c375490..7cdba8cf2b5 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -24,7 +24,7 @@ project in the group will be available to every logged in user. However, private projects will be available only if the user is a member of the project. NOTE: **Note:** -Projects of nested subgroups of a selected template source cannot be used. +Only direct subgroups can be set as the template source. Projects of nested subgroups of a selected template source cannot be used. Repository and database information that are copied over to each new project are identical to the data exported with [GitLab's Project Import/Export](../project/settings/import_export.md). diff --git a/doc/user/group/epics/img/bulk_editing.png b/doc/user/group/epics/img/bulk_editing.png Binary files differnew file mode 100644 index 00000000000..85610bef83e --- /dev/null +++ b/doc/user/group/epics/img/bulk_editing.png diff --git a/doc/user/group/epics/img/button_reopen_epic.png b/doc/user/group/epics/img/button_reopen_epic.png Binary files differindex 7a953189270..9d1be88549d 100644 --- a/doc/user/group/epics/img/button_reopen_epic.png +++ b/doc/user/group/epics/img/button_reopen_epic.png diff --git a/doc/user/group/epics/img/containing_epic.png b/doc/user/group/epics/img/containing_epic.png Binary files differindex 5ed693e6d6a..dc13d55e2bc 100644 --- a/doc/user/group/epics/img/containing_epic.png +++ b/doc/user/group/epics/img/containing_epic.png diff --git a/doc/user/group/epics/img/epic_view.png b/doc/user/group/epics/img/epic_view.png Binary files differindex dbda98e4351..c55d302ec29 100644 --- a/doc/user/group/epics/img/epic_view.png +++ b/doc/user/group/epics/img/epic_view.png diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 601ffd4947b..5968b91c9b7 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -97,6 +97,22 @@ have a [start or due date](#start-date-and-due-date), then you can see a Drag and drop to reorder issues and child epics. New issues and child epics added to an epic appear at the top of the list. +## Updating epics + +### Using bulk editing + +To apply labels across multiple epics: + +1. Go to the Epics list. +1. Click **Edit epics**. + - Checkboxes will appear beside each epic. + - A sidebar on the right-hand side will appear, with an editable field for labels. +1. Check the checkbox beside each epic to be edited. +1. Select the desired labels. +1. Click **Update all**. + + + ## Deleting an epic NOTE: **Note:** @@ -205,12 +221,12 @@ You may also consult the [group permissions table][permissions]. These text fields also fully support [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). -## Comment, or start a discussion +## Comment, or start a thread Once you wrote your comment, you can either: - Click "Comment" and your comment will be published. -- Click "Start discussion": start a thread within that epic's thread to discuss specific points. +- Click "Start thread": start a thread within that epic's discussion to discuss specific points. ## Award emoji diff --git a/doc/user/group/img/group_file_template_dropdown.png b/doc/user/group/img/group_file_template_dropdown.png Binary files differindex d9caae1a223..f0586772218 100644 --- a/doc/user/group/img/group_file_template_dropdown.png +++ b/doc/user/group/img/group_file_template_dropdown.png diff --git a/doc/user/group/img/group_file_template_settings.png b/doc/user/group/img/group_file_template_settings.png Binary files differindex ca42f7726db..5e07974bc86 100644 --- a/doc/user/group/img/group_file_template_settings.png +++ b/doc/user/group/img/group_file_template_settings.png diff --git a/doc/user/group/index.md b/doc/user/group/index.md index db348c678eb..864f1a397d5 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -78,6 +78,10 @@ Issues and merge requests are part of projects. For a given group, you can view [issues](../project/issues/index.md#issues-list) and [merge requests](../project/merge_requests/index.md#merge-requests-per-group) across all projects in that group, together in a single list view. +### Bulk editing issues and merge requests + +For details, see [bulk editing issues and merge requests](../group/bulk_editing/index.md). + ## Create a new group > For a list of words that are not allowed to be used as group names see the @@ -87,11 +91,11 @@ To create a new Group, either: - In the top menu, click **Groups** and then **Your Groups**, and click the green button **New group**. -  +  - Or, in the top menu, expand the `plus` sign and choose **New group**. -  +  Add the following information: @@ -100,15 +104,15 @@ Add the following information: 1. The **Group name** will automatically populate the URL. Optionally, you can change it. This is the name that displays in group views. The name can contain only: - - Alphanumeric characters - - Underscores - - Dashes and dots - - Spaces + - Alphanumeric characters + - Underscores + - Dashes and dots + - Spaces 1. The **Group URL** is the namespace under which your projects will be hosted. The URL can contain only: - - Alphanumeric characters - - Underscores - - Dashes and dots (it cannot start with dashes or end in a dot) + - Alphanumeric characters + - Underscores + - Dashes and dots (it cannot start with dashes or end in a dot) 1. Optionally, you can add a brief description to tell others what this group is about. 1. Optionally, choose an avatar for your group. @@ -162,12 +166,12 @@ There are two different ways to add a new project to a group: - Select a group, and then click **New project**. You can then continue [creating your project](../../gitlab-basics/create-project.md). -  +  - While you are creating a project, select a group namespace you've already created from the dropdown menu. -  +  ### Default project-creation level @@ -327,7 +331,7 @@ This will disable the option for all users who previously had permissions to operate project memberships, so no new users can be added. Furthermore, any request to add a new user to a project through API will not be possible. -#### IP access restriction **(ULTIMATE)** +#### IP access restriction **(ULTIMATE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1985) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. @@ -338,7 +342,7 @@ underlying projects, issues, etc, by IP address. This can help ensure that particular content doesn't leave the premises, while not blocking off access to the entire instance. -Add whitelisted IP subnet using CIDR notation to the group settings and anyone +Add one or more whitelisted IP subnets using CIDR notation in comma separated format to the group settings and anyone coming from a different IP address won't be able to access the restricted content. @@ -346,6 +350,38 @@ Restriction currently applies to UI, API access is not restricted. To avoid accidental lock-out, admins and group owners are are able to access the group regardless of the IP restriction. +#### Allowed domain restriction **(PREMIUM ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7297) in +[GitLab Premium](https://about.gitlab.com/pricing/) 12.2. + +You can restrict access to groups and their underlying projects by +allowing only users with email addresses in particular domains to be added to the group. + +Add email domains you want to whitelist and users with emails from different +domains won't be allowed to be added to this group. + +Some domains cannot be restricted. These are the most popular public email domains, such as: + +- `gmail.com` +- `yahoo.com` +- `hotmail.com` +- `aol.com` +- `msn.com` +- `hotmail.co.uk` +- `hotmail.fr` +- `live.com` +- `outlook.com` +- `icloud.com` + +To enable this feature: + +1. Navigate to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section, and enter domain name into **Restrict membership by email** field. +1. Click **Save changes**. + +This will enable the domain-checking for all new users added to the group from this moment on. + #### Group file templates **(PREMIUM)** Group file templates allow you to share a set of templates for common file @@ -375,6 +411,17 @@ To enable this feature, navigate to the group settings page, expand the Define project templates at a group level by setting a group as the template source. [Learn more about group-level project templates](custom_project_templates.md). +#### Disabling email notifications + +You can disable all email notifications related to the group, which also includes +it's subgroups and projects. + +To enable this feature: + +1. Navigate to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section, and select **Disable email notifications**. +1. Click **Save changes**. + ### Advanced settings - **Projects**: View all projects within that group, add members to each project, diff --git a/doc/user/group/insights/img/insights_example_stacked_bar_chart.png b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png Binary files differindex 791d0e4bcdf..0e338b99e4c 100644 --- a/doc/user/group/insights/img/insights_example_stacked_bar_chart.png +++ b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png diff --git a/doc/user/group/insights/img/insights_group_configuration.png b/doc/user/group/insights/img/insights_group_configuration.png Binary files differindex 0af0073e448..d181a1e94c3 100644 --- a/doc/user/group/insights/img/insights_group_configuration.png +++ b/doc/user/group/insights/img/insights_group_configuration.png diff --git a/doc/user/group/insights/img/insights_sidebar_link.png b/doc/user/group/insights/img/insights_sidebar_link.png Binary files differindex 64bbd1fc4d3..f7b0c2daae3 100644 --- a/doc/user/group/insights/img/insights_sidebar_link.png +++ b/doc/user/group/insights/img/insights_sidebar_link.png diff --git a/doc/user/group/roadmap/img/roadmap_view.png b/doc/user/group/roadmap/img/roadmap_view.png Binary files differindex ff41a2e0441..2be3849ba1b 100644 --- a/doc/user/group/roadmap/img/roadmap_view.png +++ b/doc/user/group/roadmap/img/roadmap_view.png diff --git a/doc/user/group/saml_sso/img/group_saml_configuration_information.png b/doc/user/group/saml_sso/img/group_saml_configuration_information.png Binary files differindex 98b83d0cb0f..e03c50ceb01 100644 --- a/doc/user/group/saml_sso/img/group_saml_configuration_information.png +++ b/doc/user/group/saml_sso/img/group_saml_configuration_information.png diff --git a/doc/user/group/saml_sso/img/group_saml_settings.png b/doc/user/group/saml_sso/img/group_saml_settings.png Binary files differindex d95acb5075f..487be4faeba 100644 --- a/doc/user/group/saml_sso/img/group_saml_settings.png +++ b/doc/user/group/saml_sso/img/group_saml_settings.png diff --git a/doc/user/group/saml_sso/img/scim_advanced.png b/doc/user/group/saml_sso/img/scim_advanced.png Binary files differindex 3b70e3fbe83..c9e095dc89a 100644 --- a/doc/user/group/saml_sso/img/scim_advanced.png +++ b/doc/user/group/saml_sso/img/scim_advanced.png diff --git a/doc/user/group/saml_sso/img/scim_attribute_mapping.png b/doc/user/group/saml_sso/img/scim_attribute_mapping.png Binary files differindex c9f6b71f5b0..933d8fb6f36 100644 --- a/doc/user/group/saml_sso/img/scim_attribute_mapping.png +++ b/doc/user/group/saml_sso/img/scim_attribute_mapping.png diff --git a/doc/user/group/saml_sso/img/scim_name_identifier_mapping.png b/doc/user/group/saml_sso/img/scim_name_identifier_mapping.png Binary files differnew file mode 100644 index 00000000000..f9c63970f16 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_name_identifier_mapping.png diff --git a/doc/user/group/saml_sso/img/scim_provisioning_status.png b/doc/user/group/saml_sso/img/scim_provisioning_status.png Binary files differnew file mode 100644 index 00000000000..41466ec9276 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_provisioning_status.png diff --git a/doc/user/group/saml_sso/img/scim_token.png b/doc/user/group/saml_sso/img/scim_token.png Binary files differindex 7eb52bf6ea2..bf9347716e9 100644 --- a/doc/user/group/saml_sso/img/scim_token.png +++ b/doc/user/group/saml_sso/img/scim_token.png diff --git a/doc/user/group/saml_sso/img/unlink_group_saml.png b/doc/user/group/saml_sso/img/unlink_group_saml.png Binary files differindex 0561443b5f4..9d53a9bf407 100644 --- a/doc/user/group/saml_sso/img/unlink_group_saml.png +++ b/doc/user/group/saml_sso/img/unlink_group_saml.png diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 54923ab69ff..bd50367681e 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -39,6 +39,25 @@ However, users will not be prompted to log via SSO on each visit. GitLab will ch We intend to add a similar SSO requirement for [Git and API activity](https://gitlab.com/gitlab-org/gitlab-ee/issues/9152) in the future. +#### Group-managed accounts + +[Introduced in GitLab 12.1](https://gitlab.com/groups/gitlab-org/-/epics/709). + +When SSO is being enforced, groups can enable an additional level of protection by enforcing the creation of dedicated user accounts to access the group. + +Without group-managed accounts, users can link their SAML identity with any existing user on the instance. With group-managed accounts enabled, users are required to create a new, dedicated user linked to the group. The notification email address associated with the user is locked to the email address received from the configured identity provider. + +When this option is enabled: + +- All existing and new users in the group will be required to log in via the SSO URL associated with the group. +- On successfully authenticating, GitLab will prompt the user to create a new, dedicated account using the email address received from the configured identity provider. +- After the group managed account has been created, group activity will require the use of this user account. + +Since use of the group managed account requires the use of SSO, users of group managed accounts will lose access to these accounts when they are no longer able to authenticate with the connected identity provider. In the case of an offboarded employee who has been removed from your identity provider: + +- The user will be unable to access the group (their credentials will no longer work on the identity provider when prompted to SSO). +- Contributions in the group (e.g. issues, merge requests) will remain intact. + ### NameID GitLab.com uses the SAML NameID to identify users. The NameID element: @@ -71,7 +90,7 @@ Once you've set up your identity provider to work with GitLab, you'll need to co 1. Navigate to the group's **Settings > SAML SSO**. 1. Find the SSO URL from your Identity Provider and enter it the **Identity provider single sign on URL** field. 1. Find and enter the fingerprint for the SAML token signing certificate in the **Certificate** field. -1. Check the **Enable SAML authentication for this group** checkbox. +1. Click the **Enable SAML authentication for this group** toggle switch. 1. Click the **Save changes** button.  diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 2d408766db8..5d136ad62da 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -27,25 +27,25 @@ The following identity providers are supported: - [Group SSO](index.md) needs to be configured. - The `scim_group` feature flag must be enabled: - Run the following commands in a Rails console: + Run the following commands in a Rails console: - ```sh - # Omnibus GitLab - gitlab-rails console + ```sh + # Omnibus GitLab + gitlab-rails console - # Installation from source - cd /home/git/gitlab - sudo -u git -H bin/rails console RAILS_ENV=production - ``` + # Installation from source + cd /home/git/gitlab + sudo -u git -H bin/rails console RAILS_ENV=production + ``` - To enable SCIM for a group named `group_name`: + To enable SCIM for a group named `group_name`: - ```ruby - group = Group.find_by_full_path('group_name') - Feature.enable(:group_scim, group) - ``` + ```ruby + group = Group.find_by_full_path('group_name') + Feature.enable(:group_scim, group) + ``` -### GitLab configuration +## GitLab configuration Once [Single sign-on](index.md) has been configured, we can: @@ -55,68 +55,84 @@ Once [Single sign-on](index.md) has been configured, we can:  -## SCIM IdP configuration +## Identity Provider configuration -### Configuration on Azure +### Azure -In the [Single sign-on](index.md) configuration for the group, make sure -that the **Name identifier value** (NameID) points to a unique identifier, such -as the `user.objectid`. This will match the `extern_uid` used on GitLab. +The SAML application that was created during [Single sign-on](index.md) setup now needs to be set up for SCIM. -The GitLab app in Azure needs to be configured following -[Azure's SCIM setup](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#getting-started). +1. Check the configuration for your GitLab SAML app and ensure that **Name identifier value** (NameID) points to `user.objectid` or another unique identifier. This will match the `extern_uid` used on GitLab. -Note the following: +  + +1. Set up automatic provisioning and administrative credentials by following the + [Provisioning users and groups to applications that support SCIM](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim) section in Azure's SCIM setup documentation. + +During this configuration, note the following: - The `Tenant URL` and `secret token` are the ones retrieved in the [previous step](#gitlab-configuration). - Should there be any problems with the availability of GitLab or similar errors, the notification email set will get those. +- It is recommended to set a notification email and check the **Send an email notification when a failure occurs** checkbox. - For mappings, we will only leave `Synchronize Azure Active Directory Users to AppName` enabled. -You can then test the connection clicking on `Test Connection`. +You can then test the connection by clicking on **Test Connection**. If the connection is successful, be sure to save your configuration before moving on. See below for [troubleshooting](#troubleshooting). + +#### Configure attribute mapping -### Synchronize Azure Active Directory users +1. Click on `Synchronize Azure Active Directory Users to AppName`, to configure the attribute mapping. +1. Click **Delete** next to the `mail` mapping. +1. Map `userPrincipalName` to `emails[type eq "work"].value` and change it's **Matching precedence** to `2`. +1. Map `mailNickname` to `userName`. +1. Create a new mapping by clicking **Add New Mapping** then set **Source attribute** to `objectId`, **Target attribute** to `id`, **Match objects using this attribute** to `Yes`, and **Matching precedence** to `1`. +1. Create a new mapping by clicking **Add New Mapping** then set **Source attribute** to `objectId`, and **Target attribute** to `externalId`. +1. Click the `userPrincipalName` mapping and change **Match objects using this attribute** to `No`. -1. Click on `Synchronize Azure Active Directory Users to AppName`, to configure - the attribute mapping. -1. Select the unique identifier (in the example `objectId`) as the `id` and `externalId`, - and enable the `Create`, `Update`, and `Delete` actions. -1. Map the `userPricipalName` to `emails[type eq "work"].value` and `mailNickname` to - `userName`. + Save your changes and you should have the following configuration: - Example configuration: +  -  + NOTE: **Note:** If you used a unique identifier **other than** `objectId`, be sure to map it instead to both `id` and `externalId`. + +1. Below the mapping list click on **Show advanced options > Edit attribute list for AppName**. -1. Click on **Show advanced options > Edit attribute list for AppName**. 1. Leave the `id` as the primary and only required field. - NOTE: **Note:** - `username` should neither be primary nor required as we don't support - that field on GitLab SCIM yet. + NOTE: **Note:** + `username` should neither be primary nor required as we don't support + that field on GitLab SCIM yet. -  +  1. Save all the screens and, in the **Provisioning** step, set - the `Provisioning Status` to `ON`. + the `Provisioning Status` to `On`. + +  - NOTE: **Note:** - You can control what is actually synced by selecting the `Scope`. For example, - `Sync only assigned users and groups` will only sync the users assigned to - the application (`Users and groups`), otherwise it will sync the whole Active Directory. + NOTE: **Note:** + You can control what is actually synced by selecting the `Scope`. For example, + `Sync only assigned users and groups` will only sync the users assigned to + the application (`Users and groups`), otherwise, it will sync the whole Active Directory. Once enabled, the synchronization details and any errors will appear on the bottom of the **Provisioning** screen, together with a link to the audit logs. -<!-- ## Troubleshooting +## Troubleshooting + +### Testing Azure connection: invalid credentials + +When testing the connection, you may encounter an error: **You appear to have entered invalid credentials. Please confirm you are using the correct information for an administrative account**. If `Tenant URL` and `secret token` are correct, check whether your group path contains characters that may be considered invalid JSON primitives (such as `.`). Removing such characters from the group path typically resolves the error. + +### Azure: (Field) can't be blank sync error + +When checking the Audit Logs for the Provisioning, you can sometimes see the +error `Namespace can't be blank, Name can't be blank, and User can't be blank.` + +This is likely caused because not all required fields (such as first name and last name) are present for all users being mapped. -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +As a workaround, try an alternate mapping: -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +1. Follow the Azure mapping instructions from above. +1. Delete the `name.formatted` target attribute entry. +1. Change the `displayName` source attribute to have `name.formatted` target attribute. diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 1c6cca049c5..2eb3fae1a85 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -28,39 +28,39 @@ only 1 parent group. It resembles a directory behavior or a nested items list: - Group 1 - Group 1.1 - Group 1.2 - - Group 1.2.1 - - Group 1.2.2 - - Group 1.2.2.1 + - Group 1.2.1 + - Group 1.2.2 + - Group 1.2.2.1 In a real world example, imagine maintaining a GNU/Linux distribution with the first group being the name of the distribution, and subsequent groups split as follows: - Organization Group - GNU/Linux distro - Category Subgroup - Packages - - (project) Package01 - - (project) Package02 + - (project) Package01 + - (project) Package02 - Category Subgroup - Software - - (project) Core - - (project) CLI - - (project) Android app - - (project) iOS app + - (project) Core + - (project) CLI + - (project) Android app + - (project) iOS app - Category Subgroup - Infra tools - - (project) Ansible playbooks + - (project) Ansible playbooks Another example of GitLab as a company would be the following: - Organization Group - GitLab - Category Subgroup - Marketing - - (project) Design - - (project) General + - (project) Design + - (project) General - Category Subgroup - Software - - (project) GitLab CE - - (project) GitLab EE - - (project) Omnibus GitLab - - (project) GitLab Runner - - (project) GitLab Pages daemon + - (project) GitLab CE + - (project) GitLab EE + - (project) Omnibus GitLab + - (project) GitLab Runner + - (project) GitLab Pages daemon - Category Subgroup - Infra tools - - (project) Chef cookbooks + - (project) Chef cookbooks - Category Subgroup - Executive team --- @@ -74,27 +74,37 @@ structure. ## Creating a subgroup -NOTE: **Note:** -You must be an Owner of a group to create a subgroup. For -more information check the [permissions table](../../permissions.md#group-members-permissions). -For a list of words that are not allowed to be used as group names see the +To create a subgroup you must either be an Owner or a Maintainer of the +group, depending on the group's setting. + +By default, groups created in: + +- GitLab 12.2 or later allow both Owners and Maintainers to create subgroups. +- GitLab 12.1 or earlier only allow Owners to create subgroups. + +This setting can be for any group by an Owner or Administrator. + +For more information check the +[permissions table](../../permissions.md#group-members-permissions). For a list +of words that are not allowed to be used as group names see the [reserved names](../../reserved_names.md). -Users can always create subgroups if they are explicitly added as an Owner to -a parent group, even if group creation is disabled by an administrator in their -settings. + +Users can always create subgroups if they are explicitly added as an Owner (or +Maintainer, if that setting is enabled) to a parent group, even if group +creation is disabled by an administrator in their settings. To create a subgroup: 1. In the group's dashboard expand the **New project** split button, select **New subgroup** and click the **New subgroup** button. -  +  1. Create a new group like you would normally do. Notice that the parent group namespace is fixed under **Group path**. The visibility level can differ from the parent group. -  +  1. Click the **Create group** button and you will be taken to the new group's dashboard page. diff --git a/doc/user/index.md b/doc/user/index.md index 501d74c76d1..27e75189fc3 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -2,7 +2,7 @@ description: 'Read through the GitLab User documentation to learn how to use, configure, and customize GitLab and GitLab.com to your own needs.' --- -# User documentation +# User Docs Welcome to GitLab! We're glad to have you here! @@ -53,7 +53,7 @@ With GitLab Enterprise Edition, you can also: - Improve collaboration with [Merge Request Approvals](project/merge_requests/index.md#merge-request-approvals-starter), [Multiple Assignees for Issues](project/issues/multiple_assignees_for_issues.md), - and [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards-starter). + and [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards). - Create formal relationships between issues with [Related Issues](project/issues/related_issues.md). - Use [Burndown Charts](project/milestones/burndown_charts.md) to track progress during a sprint or while working on a new version of their software. - Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Global Search](search/advanced_global_search.md) and [Advanced Syntax Search](search/advanced_search_syntax.md) for faster, more advanced code search across your entire GitLab instance. @@ -124,12 +124,12 @@ merge requests, code snippets, and commits. When performing inline reviews to implementations to your codebase through merge requests you can -gather feedback through [resolvable discussions](discussions/index.md#resolvable-comments-and-discussions). +gather feedback through [resolvable threads](discussions/index.md#resolvable-comments-and-threads). ### GitLab Flavored Markdown (GFM) Read through the [GFM documentation](markdown.md) to learn how to apply -the best of GitLab Flavored Markdown in your discussions, comments, +the best of GitLab Flavored Markdown in your threads, comments, issues and merge requests descriptions, and everywhere else GMF is supported. @@ -148,7 +148,7 @@ requests you're assigned to. [Snippets](snippets.md) are code blocks that you want to store in GitLab, from which you have quick access to. You can also gather feedback on them through -[discussions](#discussions). +[Discussions](#Discussions). ## Integrations diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 894f83d3c75..f557dcf4b3c 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -1,7 +1,6 @@ # Instance-level Kubernetes clusters > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/39840) in GitLab 11.11. -> Instance-level cluster integration is currently in [Beta](https://about.gitlab.com/handbook/product/#alpha-beta-ga). ## Overview diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 83629fc2e4b..17bbed2945d 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -40,7 +40,7 @@ repositories are also processed with CommonMark. As of 11.8, the [Redcarpet Ruby has been removed and all issues and comments, including those from pre-11.1, are now processed using the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker). -The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) +The documentation website had its [markdown engine migrated from Redcarpet to Kramdown](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/108) in October 2018. You may have older issues, merge requests, or Markdown documents in your @@ -57,12 +57,6 @@ render incorrectly: - milk ``` -1. Chocolate - - dark - - milk - ---- - Simply add a space to each nested item to align the `-` with the first character of the top list item (`C` in this case): @@ -136,26 +130,26 @@ Supported formats (named colors are not supported): Color written inside backticks will be followed by a color "chip": ```markdown -`#F00` -`#F00A` -`#FF0000` -`#FF0000AA` -`RGB(0,255,0)` -`RGB(0%,100%,0%)` -`RGBA(0,255,0,0.3)` -`HSL(540,70%,50%)` -`HSLA(540,70%,50%,0.3)` -``` - -`#F00` -`#F00A` -`#FF0000` -`#FF0000AA` -`RGB(0,255,0)` -`RGB(0%,100%,0%)` -`RGBA(0,255,0,0.3)` -`HSL(540,70%,50%)` -`HSLA(540,70%,50%,0.3)` +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.3)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.3)` +``` + +`#F00` +`#F00A` +`#FF0000` +`#FF0000AA` +`RGB(0,255,0)` +`RGB(0%,100%,0%)` +`RGBA(0,255,0,0.3)` +`HSL(540,70%,50%)` +`HSLA(540,70%,50%,0.3)` ### Diagrams and flowcharts using Mermaid @@ -185,6 +179,49 @@ graph TD; C-->D; ``` +#### Subgraphs + +NOTE: **Note:** GitLab 12.1 and up now [requires quotes around subgraph +titles that contain multiple words](https://github.com/knsv/mermaid/pull/845). + +Subgraphs can also be included: + +~~~ +```mermaid +graph TB + + SubGraph1 --> SubGraph1Flow + subgraph "SubGraph 1 Flow" + SubGraph1Flow(SubNode 1) + SubGraph1Flow -- Choice1 --> DoChoice1 + SubGraph1Flow -- Choice2 --> DoChoice2 + end + + subgraph "Main Graph" + Node1[Node 1] --> Node2[Node 2] + Node2 --> SubGraph1[Jump to SubGraph1] + SubGraph1 --> FinalThing[Final Thing] +end +``` +~~~ + +```mermaid +graph TB + + SubGraph1 --> SubGraph1Flow + subgraph "SubGraph 1 Flow" + SubGraph1Flow(SubNode 1) + SubGraph1Flow -- Choice1 --> DoChoice1 + SubGraph1Flow -- Choice2 --> DoChoice2 + end + + subgraph "Main Graph" + Node1[Node 1] --> Node2[Node 2] + Node2 --> SubGraph1[Jump to SubGraph1] + SubGraph1 --> FinalThing[Final Thing] +end +``` + ### Emoji > If this is not rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji). @@ -222,8 +259,7 @@ this font installed by default. ### Front matter -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) - in GitLab 11.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23331) in GitLab 11.6. Front matter is metadata included at the beginning of a markdown document, preceding its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/), @@ -397,6 +433,7 @@ unordered or ordered lists: - [ ] Sub-task 1 - [x] Sub-task 2 - [ ] Sub-task 3 + 1. [x] Completed task 1. [ ] Incomplete task 1. [ ] Sub-task 1 @@ -408,6 +445,7 @@ unordered or ordered lists: - [ ] Sub-task 1 - [x] Sub-task 2 - [ ] Sub-task 3 + 1. [x] Completed task 1. [ ] Incomplete task 1. [ ] Sub-task 1 @@ -488,6 +526,10 @@ This snippet links to `<wiki_root>/miscellaneous.md`: [Link to Related Page](/miscellaneous.md) ``` +### Embedding metrics in GitLab Flavored Markdown + +Metric charts can be embedded within GitLab Flavored Markdown. See [Embedding Metrics within GitLab flavored Markdown](../user/project/integrations/prometheus.md#embedding-metric-charts-within-gitlab-flavored-markdown) for more details. + ## Standard markdown and extensions in GitLab All standard markdown formatting should work as expected within GitLab. Some standard @@ -556,7 +598,7 @@ Inline `code` has `back-ticks around` it. Similarly, a whole block of code can be fenced with triple backticks ```` ``` ````, triple tildes (`~~~`), or indended 4 or more spaces to achieve a similar effect for -a larger body of code. test. +a larger body of code. ~~~ ``` @@ -586,9 +628,11 @@ def function(): print s ``` - Using 4 spaces - is like using - 3-backtick fences. +``` +Using 4 spaces +is like using +3-backtick fences. +``` ~~~ Tildes are OK too. @@ -815,18 +859,6 @@ or underscores ___ ``` -Three or more hyphens, - ---- - -asterisks, - -*** - -or underscores - -___ - ### Images Examples: @@ -974,7 +1006,7 @@ after the `</summary>` tag and before the `</details>` tag, as shown in the exam These details _will_ remain **hidden** until expanded. - PASTE LOGS HERE +PASTE LOGS HERE </details> ``` @@ -986,7 +1018,7 @@ These details _will_ remain **hidden** until expanded. These details <em>will</em> remain <b>hidden</b> until expanded. - PASTE LOGS HERE +PASTE LOGS HERE </details> @@ -1045,14 +1077,14 @@ A new line due to the previous backslash. First paragraph. Another line in the same paragraph. -A third line in the same paragraph, but this time ending with two spaces. +A third line in the same paragraph, but this time ending with two spaces. A new line directly under the first paragraph. <!-- (Do *NOT* remove the two ending whitespaces in the second line) --> <!-- (They are needed for the Markdown text to render correctly on docs.gitlab.com, the backslash works fine inside GitLab itself) --> Second paragraph. -Another line, this time ending with a backslash. +Another line, this time ending with a backslash. A new line due to the previous backslash. ### Links @@ -1133,13 +1165,13 @@ GFM will autolink almost any URL you put into your text: ### Lists Ordered and unordered lists can be easily created. Add the number you want the list -to start with, like `1. ` (with a space) at the start of each line for ordered lists. +to start with, like `1.`, followed by a space, at the start of each line for ordered lists. After the first number, it does not matter what number you use, ordered lists will be -numbered automatically by vertical order, so repeating `1. ` for all items in the -same list is common. If you start with a number other than `1. `, it will use that as the first +numbered automatically by vertical order, so repeating `1.` for all items in the +same list is common. If you start with a number other than `1.`, it will use that as the first number, and count up from there. -Add a `* `, `- ` or `+ ` (with a space) at the start of each line for unordered lists, but +Add a `*`, `-` or `+`, followed by a space, at the start of each line for unordered lists, but you should not use a mix of them. Examples: @@ -1154,21 +1186,27 @@ Examples: 4. And another item. * Unordered lists can use asterisks + - Or minuses + + Or pluses ``` +<!-- The "2." and "4." in the example above are changed to "1." below, only to match the standards on docs.gitlab.com --> + 1. First ordered list item -2. Another item +1. Another item - Unordered sub-list. 1. Actual numbers don't matter, just that it's a number 1. Ordered sub-list 1. Next ordered sub-list item -4. And another item. +1. And another item. + +- Unordered lists can use asterisks -* Unordered lists can use asterisks - Or minuses -+ Or pluses + +- Or pluses --- @@ -1182,14 +1220,14 @@ Example: Second paragraph of first item. -2. Another item +1. Another item ``` 1. First ordered list item Second paragraph of first item. -2. Another item +1. Another item --- @@ -1203,14 +1241,14 @@ Example: Paragraph of first item. -2. Another item +1. Another item ``` 1. First ordered list item Paragraph of first item. -2. Another item +1. Another item ### Superscripts / Subscripts diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png Binary files differindex 9d6a509ea72..d4db5e88672 100644 --- a/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png Binary files differindex ee33eee8134..7dbec81c18a 100644 --- a/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 1b279173d1c..80d1bf992ec 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -45,8 +45,10 @@ The following table depicts the various user permission levels in a project. | Leave comments | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View Insights charts **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View approved/blacklisted licenses **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View license management reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View License Compliance reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View Security reports **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View Dependency list **(ULTIMATE)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View [Design Management](project/issues/design_management.md) pages **(PREMIUM)** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | Pull project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View GitLab Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control-core-only) | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -60,10 +62,9 @@ The following table depicts the various user permission levels in a project. | View confidential issues | (*2*) | ✓ | ✓ | ✓ | ✓ | | Assign issues | | ✓ | ✓ | ✓ | ✓ | | Label issues | | ✓ | ✓ | ✓ | ✓ | -| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | +| Lock issue threads | | ✓ | ✓ | ✓ | ✓ | | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | | Manage related issues **(STARTER)** | | ✓ | ✓ | ✓ | ✓ | -| Create issue from vulnerability **(ULTIMATE)** | | ✓ | ✓ | ✓ | ✓ | | Manage labels | | ✓ | ✓ | ✓ | ✓ | | Create code snippets | | ✓ | ✓ | ✓ | ✓ | | See a commit status | | ✓ | ✓ | ✓ | ✓ | @@ -74,6 +75,7 @@ The following table depicts the various user permission levels in a project. | View Error Tracking list | | ✓ | ✓ | ✓ | ✓ | | Pull from [Maven repository](project/packages/maven_repository.md) or [NPM registry](project/packages/npm_registry.md) **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | | Publish to [Maven repository](project/packages/maven_repository.md) or [NPM registry](project/packages/npm_registry.md) **(PREMIUM)** | | | ✓ | ✓ | ✓ || +| Upload [Design Management](project/issues/design_management.md) files **(PREMIUM)** | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | | Push to non-protected branches | | | ✓ | ✓ | ✓ | | Force push to non-protected branches | | | ✓ | ✓ | ✓ | @@ -81,7 +83,7 @@ The following table depicts the various user permission levels in a project. | Create new merge request | | | ✓ | ✓ | ✓ | | Assign merge requests | | | ✓ | ✓ | ✓ | | Label merge requests | | | ✓ | ✓ | ✓ | -| Lock merge request discussions | | | ✓ | ✓ | ✓ | +| Lock merge request threads | | | ✓ | ✓ | ✓ | | Manage/Accept merge requests | | | ✓ | ✓ | ✓ | | Create new environments | | | ✓ | ✓ | ✓ | | Stop environments | | | ✓ | ✓ | ✓ | @@ -92,6 +94,8 @@ The following table depicts the various user permission levels in a project. | Remove a container registry image | | | ✓ | ✓ | ✓ | | Create/edit/delete project milestones | | | ✓ | ✓ | ✓ | | Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| View vulnerabilities in Dependency list **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| Create issue from vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | Dismiss vulnerability **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | Apply code change suggestions | | | ✓ | ✓ | ✓ | | Create and edit wiki pages | | | ✓ | ✓ | ✓ | @@ -122,6 +126,7 @@ The following table depicts the various user permission levels in a project. | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | | Delete issues | | | | | ✓ | +| Disable notification emails | | | | | ✓ | | Force push to protected branches (*4*) | | | | | | | Remove protected branches (*4*) | | | | | | @@ -157,7 +162,7 @@ to learn more. ### Cycle Analytics permissions Find the current permissions on the Cycle Analytics dashboard on -the [documentation on Cycle Analytics permissions](project/cycle_analytics.md#permissions). +the [documentation on Cycle Analytics permissions](analytics/cycle_analytics.md#permissions). ### Issue Board permissions @@ -209,12 +214,17 @@ group. | Create project in group | | | ✓ | ✓ | ✓ | | Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | | Enable/disable a dependency proxy **(PREMIUM)** | | | ✓ | ✓ | ✓ | +| Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | +| Create subgroup | | | | ✓ (1) | ✓ | | Edit group | | | | | ✓ | -| Create subgroup | | | | | ✓ | | Manage group members | | | | | ✓ | | Remove group | | | | | ✓ | | Delete group epic **(ULTIMATE)** | | | | | ✓ | | View group Audit Events | | | | | ✓ | +| Disable notification emails | | | | | ✓ | + +- (1): Groups can be set to [allow either Owners or Owners and + Maintainers to create subgroups](group/subgroups/index.md#creating-a-subgroup) ### Subgroup permissions @@ -225,6 +235,19 @@ nested groups if you have membership in one of its parents. To learn more, read through the documentation on [subgroups memberships](group/subgroups/index.md#membership). +## Guest User + +When a user is given `Guest` permissions on a project and/or group, and holds no +higher permission level on any other project or group on the instance, the user +is considered a guest user by GitLab and will not consume a license seat. +There is no other specific "guest" designation for newly created users. + +If the user is assigned a higher role on any projects or groups, the user will +take a license seat. If a user creates a project, the user becomes a `Maintainer` +on the project, resulting in the use of a license seat. To prevent a guest user +from creating projects, you can edit the user profile to mark the user as +[External](#external-users-permissions). + ## External users permissions In cases where it is desired that a user has access only to some internal or diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index 304a7984191..5b0954195f8 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -1,37 +1,87 @@ -# Deleting a User Account +--- +type: howto +--- + +# Deleting a User account + +Users can be deleted from a GitLab instance, either by: + +- The user themselves. +- An administrator. NOTE: **Note:** Deleting a user will delete all projects in that user namespace. -- As a user, you can delete your own account by navigating to **Settings** > **Account** and selecting **Delete account** -- As an admin, you can delete a user account by navigating to the **Admin Area**, selecting the **Users** tab, selecting a user, and clicking on **Delete user** +## As a user + +As a user, you can delete your own account by: + +1. Clicking on your avatar. +1. Navigating to **Settings > Account**. +1. Selecting **Delete account**. + +## As an administrator + +As an administrator, you can delete a user account by: + +1. Navigating to **Admin Area > Overview > Users**. +1. Selecting a user. +1. Under the **Account** tab, clicking: + - **Delete user** to delete only the user but maintaining their + [associated records](#associated-records). + - **Delete user and contributions** to delete the user and + their associated records. + +### Blocking a user + +In addition to blocking a user +[via an abuse report](../../admin_area/abuse_reports.md#blocking-users), +a user can be blocked directly from the Admin area. To do this: + +1. Navigate to **Admin Area > Overview > Users**. +1. Selecting a user. +1. Under the **Account** tab, click **Block user**. ## Associated Records -> Introduced for issues in [GitLab 9.0][ce-7393], and for merge requests, award - emoji, notes, and abuse reports in [GitLab 9.1][ce-10467]. - Hard deletion from abuse reports and spam logs was introduced in - [GitLab 9.1][ce-10273], and from the API in [GitLab 9.3][ce-11853]. +> - Introduced for issues in +> [GitLab 9.0](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7393). +> - Introduced for merge requests, award emoji, notes, and abuse reports in +> [GitLab 9.1](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10467). +> - Hard deletion from abuse reports and spam logs was introduced in +> [GitLab 9.1](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10273), +> and from the API in +> [GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11853). When a user account is deleted, not all associated records are deleted with it. Here's a list of things that will **not** be deleted: -- Issues that the user created -- Merge requests that the user created -- Notes that the user created -- Abuse reports that the user reported -- Award emoji that the user created +- Issues that the user created. +- Merge requests that the user created. +- Notes that the user created. +- Abuse reports that the user reported. +- Award emoji that the user created. Instead of being deleted, these records will be moved to a system-wide -user with the username "Ghost User", whose sole purpose is to act as a container for such records. Any commits made by a deleted user will still display the username of the original user. +user with the username "Ghost User", whose sole purpose is to act as a container +for such records. Any commits made by a deleted user will still display the +username of the original user. -When a user is deleted from an [abuse report](../../admin_area/abuse_reports.md) or spam log, these associated +When a user is deleted from an [abuse report](../../admin_area/abuse_reports.md) +or spam log, these associated records are not ghosted and will be removed, along with any groups the user -is a sole owner of. Administrators can also request this behaviour when +is a sole owner of. Administrators can also request this behavior when deleting users from the [API](../../../api/users.md#user-deletion) or the -admin area. +Admin Area. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -[ce-7393]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7393 -[ce-10273]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10273 -[ce-10467]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10467 -[ce-11853]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11853 +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index e3e8c9a0d6d..f7ba921aa7d 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -1,3 +1,7 @@ +--- +type: howto +--- + # Two-Factor Authentication Two-factor Authentication (2FA) provides an additional level of security to your @@ -15,7 +19,7 @@ When you enable 2FA, don't forget to back up your [recovery codes](#recovery-cod In addition to time-based one time passwords (TOTP), GitLab supports U2F (universal 2nd factor) devices as the second factor of authentication. Once -enabled, in addition to supplying your username and password to login, you'll +enabled, in addition to supplying your username and password to log in, you'll be prompted to activate your U2F device (usually by pressing a button on it), and it will perform secure authentication on your behalf. @@ -44,6 +48,7 @@ To enable 2FA: - [andOTP](https://github.com/andOTP/andOTP): feature rich open source app for Android which supports PGP encrypted backups. - [FreeOTP](https://freeotp.github.io/): open source app for Android. - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en): proprietary app for iOS and Android. + - [SailOTP](https://openrepos.net/content/seiichiro0185/sailotp): open source app for SailFish OS. 1. In the application, add a new entry in one of two ways: - Scan the code presented in GitLab with your device's camera to add the entry automatically. @@ -183,29 +188,29 @@ a new set of recovery codes with SSH: 1. You will then be prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes: - ```sh - Are you sure you want to generate new two-factor recovery codes? - Any existing recovery codes you saved will be invalidated. (yes/no) - - yes - - Your two-factor authentication recovery codes are: - - 119135e5a3ebce8e - 11f6v2a498810dcd - 3924c7ab2089c902 - e79a3398bfe4f224 - 34bd7b74adbc8861 - f061691d5107df1a - 169bf32a18e63e7f - b510e7422e81c947 - 20dbed24c5e74663 - df9d3b9403b9c9f0 - - During sign in, use one of the codes above when prompted for your - two-factor code. Then, visit your Profile Settings and add a new device - so you do not lose access to your account again. - ``` + ```sh + Are you sure you want to generate new two-factor recovery codes? + Any existing recovery codes you saved will be invalidated. (yes/no) + + yes + + Your two-factor authentication recovery codes are: + + 119135e5a3ebce8e + 11f6v2a498810dcd + 3924c7ab2089c902 + e79a3398bfe4f224 + 34bd7b74adbc8861 + f061691d5107df1a + 169bf32a18e63e7f + b510e7422e81c947 + 20dbed24c5e74663 + df9d3b9403b9c9f0 + + During sign in, use one of the codes above when prompted for your + two-factor code. Then, visit your Profile Settings and add a new device + so you do not lose access to your account again. + ``` 1. Go to the GitLab sign-in page and enter your username/email and password. When prompted for a two-factor code, enter one of the recovery codes obtained @@ -238,3 +243,15 @@ Sign in and re-enable two-factor authentication as soon as possible. - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because the U2F key has only been registered on `first.host.xyz`. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index 28e3f4904a9..2d7bd25fc27 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -1,14 +1,31 @@ +--- +type: howto +--- + # Active Sessions > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17867) > in GitLab 10.8. GitLab lists all devices that have logged into your account. This allows you to -review the sessions. +review the sessions, and revoke any you don't recognize. ## Listing all active sessions -1. On the upper right corner, click on your avatar and go to your **Settings**. -1. Navigate to the **Active Sessions** tab. +1. Click your avatar. +1. Select **Settings**. +1. Click **Active Sessions** in the sidebar.  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/img/active_sessions_list.png b/doc/user/profile/img/active_sessions_list.png Binary files differindex 1e242ac4710..41173c7eee5 100644 --- a/doc/user/profile/img/active_sessions_list.png +++ b/doc/user/profile/img/active_sessions_list.png diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 61a30a775b0..e5ccc8ee758 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -1,7 +1,12 @@ +--- +type: index, howto +--- + # User account -When signed into their GitLab account, users can customize their -experience according to the best approach to their cases. +Each GitLab account has a user profile, and settings. Your [profile](#user-profile) +contains information about you, and your GitLab activity. Your [settings](#profile-settings) +allow you to customize some aspects of GitLab to suit yourself. ## Signing in @@ -10,8 +15,10 @@ See the [authentication topic](../../topics/authentication/index.md) for more de ## User profile -Your profile is available from the up-right corner menu bar (user's avatar) > **Profile**, -or from `https://example.gitlab.com/username`. +To access your profile: + +1. Click on your avatar. +1. Select **Profile**. On your profile page, you will see the following information: @@ -20,12 +27,15 @@ On your profile page, you will see the following information: - Groups: [groups](../group/index.md) you're a member of - Contributed projects: [projects](../project/index.md) you contributed to - Personal projects: your personal projects (respecting the project's visibility level) +- Starred projects: projects you starred - Snippets: your personal code [snippets](../snippets.md#personal-snippets) ## Profile settings -You can edit your account settings by navigating from the up-right corner menu bar -(user's avatar) > **Settings**, or visiting `https://example.gitlab.com/profile`. +To access your profile settings: + +1. Click on your avatar. +1. Select **Settings**. From there, you can: @@ -56,8 +66,8 @@ before proceeding. To change your `username`: 1. Navigate to your [profile's](#profile-settings) **Settings > Account**. -1. Enter a new username under "Change username". -1. Hit **Update username**. +1. Enter a new username under **Change username**. +1. Click **Update username**. CAUTION: **Caution:** It is currently not possible to change your username if it contains a @@ -82,16 +92,20 @@ The following information will be hidden from the user profile page (`https://gi - Groups tab - Contributed projects tab - Personal projects tab +- Starred projects tab - Snippets tab To enable private profile: -1. Navigate to your personal [profile settings](#profile-settings). -1. Check the "Private profile" option. -1. Hit **Update profile settings**. +1. Click your avatar. +1. Select **Profile**. +1. Click **Edit profile** (pencil icon). +1. Check the **Private profile** option in the **Main settings** section. +1. Click **Update profile settings**. NOTE: **Note:** -You and GitLab admins can see your the abovementioned information on your profile even if it is private. +All your profile information can be seen by yourself, and GitLab admins, even if +the **Private profile** option is enabled. ## Add details of external accounts @@ -99,9 +113,15 @@ GitLab allows you to add links to certain other external accounts you might have To add links to other accounts: -1. Navigate to your **User Settings > Profile**. -1. In the **Main settings** section, locate and fill out fields for links to external accounts like Skype and Twitter. -1. Click the **Update profile settings** button. +1. Click your avatar. +1. Select **Profile**. +1. Click **Edit profile** (pencil icon). +1. Complete the desired fields for external accounts, in the **Main settings** + section: + - Skype + - Twitter + - LinkedIn +1. Click **Update profile settings**. ## Private contributions @@ -111,9 +131,11 @@ Enabling private contributions will include contributions to private projects, i To enable private contributions: -1. Navigate to your personal [profile settings](#profile-settings). -1. Check the "Private contributions" option. -1. Hit **Update profile settings**. +1. Click on your avatar. +1. Select **Profile**. +1. Click **Edit profile** (pencil icon). +1. Check the **Private contributions** option. +1. Click **Update profile settings**. ## Current status @@ -124,22 +146,24 @@ This may be helpful when you are out of office or otherwise not available. Other users can then take your status into consideration when responding to your issues or assigning work to you. Please be aware that your status is publicly visible even if your [profile is private](#private-profile). +Status messages are restricted to 100 characters of plain text. +They may however contain emoji codes such as `I'm on vacation :palm_tree:`. + To set your current status: -1. Open the user menu in the top-right corner of the navigation bar. -1. Hit **Set status**, or **Edit status** if you have already set a status. -1. Set the emoji and/or status message to your liking. -1. Hit **Set status**. Alternatively, you can also hit **Remove status** to remove your user status entirely. +1. Click your avatar. +1. Click **Set status**, or **Edit status** if you have already set a status. +1. Set the desired emoji and/or status message. +1. Click **Set status**. Alternatively, you can click **Remove status** to remove your user status entirely. or -1. Navigate to your personal [profile settings](#profile-settings). -1. In the text field below `Your status`, enter your status message. -1. Select an emoji from the dropdown if you like. -1. Hit **Update profile settings**. - -Status messages are restricted to 100 characters of plain text. -They may however contain emoji codes such as `I'm on vacation :palm_tree:`. +1. Click your avatar. +1. Select **Profile**. +1. Click **Edit profile** (pencil icon). +1. Enter your status message in the **Your status** text field. +1. Click **Add status emoji** (smiley face), and select the desired emoji. +1. Click **Update profile settings**. You can also set your current status [using the API](../../api/users.md#user-status). @@ -147,39 +171,42 @@ You can also set your current status [using the API](../../api/users.md#user-sta > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21598) in GitLab 11.4. -A commit email, is the email that will be displayed in every Git-related action done through the -GitLab interface. +A commit email is an email address displayed in every Git-related action carried out through the GitLab interface. -You are able to select from the list of your own verified emails which email you want to use as the commit email. +Any of your own verified email addresses can be used as the commit email. -To change it: +To change your commit email: -1. Open the user menu in the top-right corner of the navigation bar. -1. Hit **Commit email** selection box. +1. Click your avatar. +1. Select **Profile**. +1. Click **Edit profile** (pencil icon). +1. Click **Commit email** dropdown. 1. Select any of the verified emails. -1. Hit **Update profile settings**. +1. Click **Update profile settings**. ### Private commit email > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22560) in GitLab 11.5. GitLab provides the user with an automatically generated private commit email option, -which allows the user to not make their email information public. +which allows the user to keep their email information private. To enable this option: -1. Open the user menu in the top-right corner of the navigation bar. -1. Hit **Commit email** selection box. +1. Click your avatar. +1. Select **Profile**. +1. Click **Edit profile** (pencil icon). +1. Click **Commit email** dropdown. 1. Select **Use a private email** option. -1. Hit **Update profile settings**. +1. Click **Update profile settings**. Once this option is enabled, every Git-related action will be performed using the private commit email. -In order to stay fully annonymous, you can also copy this private commit email +To stay fully anonymous, you can also copy this private commit email and configure it on your local machine using the following command: -``` -git config --global user.email "YOUR_PRIVATE_COMMIT_EMAIL" +```sh +git config --global user.email <your email address> ``` ## Troubleshooting @@ -202,3 +229,15 @@ to get you a new `_gitlab_session` and keep you signed in through browser restar After your `remember_user_token` expires and your `_gitlab_session` is cleared/expired, you will be asked to sign in again to verify your identity (which is for security reasons). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 0b224fc7e01..d556daa3460 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -1,3 +1,7 @@ +--- +type: concepts, howto +--- + # Personal access tokens > [Introduced][ce-3749] in GitLab 8.8. @@ -40,7 +44,7 @@ the following table. | Scope | Introduced in | Description | | ------------------ | ------------- | ----------- | | `read_user` | [GitLab 8.15](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951) | Allows access to the read-only endpoints under `/users`. Essentially, any of the `GET` requests in the [Users API][users] are allowed. | -| `api` | [GitLab 8.15](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951) | Grants complete access to the API and Container Registry (read/write). | +| `api` | [GitLab 8.15](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951) | Grants complete read/write access to the API, including all groups and projects, the container registry, and the package registry. | | `read_registry` | [GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845) | Allows to read (pull) [container registry] images if a project is private and authorization is required. | | `sudo` | [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14838) | Allows performing API actions as any user in the system (if the authenticated user is an admin). | | `read_repository` | [GitLab 10.7](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17894) | Allows read-only access (pull) to the repository through git clone. | @@ -52,3 +56,15 @@ the following table. [container registry]: ../project/container_registry.md [users]: ../../api/users.md [usage]: ../../api/README.md#personal-access-tokens + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index b1fde3b577b..82a6d2b3703 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -1,11 +1,17 @@ +--- +type: concepts, howto +--- + # Profile preferences A user's profile preferences page allows the user to customize various aspects of GitLab to their liking. -To navigate to your profile's preferences, click your avatar icon in the top -right corner, select **Settings** and then choose **Preferences** from the -left sidebar. +To navigate to your profile's preferences: + +1. Click your avatar. +1. Select **Settings**. +1. Click **Preferences** in the sidebar. ## Navigation theme @@ -15,7 +21,7 @@ and left side navigation. Using individual color themes might help you differentiate between your different GitLab instances. -The default palette is Indigo. You can choose between 10 different themes: +The default theme is Indigo. You can choose between 10 themes: - Indigo - Light Indigo @@ -39,7 +45,7 @@ for syntax highlighting. For a list of supported languages visit the rouge websi Changing this setting allows you to customize the color theme when viewing any syntax highlighted code on GitLab. -The default syntax theme is White, and you can choose among 5 different colors: +The default syntax theme is White, and you can choose among 5 different themes: - White - Dark @@ -81,6 +87,16 @@ You have 8 options here that you can use for your default dashboard view: - Assigned Merge Requests - Operations Dashboard **(PREMIUM)** +### Group overview content + +The **Group overview content** dropdown allows you to choose what information is +displayed on a group’s home page. + +You can choose between 2 options: + +- Details (default) +- [Security dashboard](../application_security/security_dashboard/index.md) **(ULTIMATE)** + ### Project overview content The project overview content setting allows you to choose what content you want to @@ -102,7 +118,7 @@ Select your preferred language from a list of supported languages. ### First day of the week -The first day of the week can be customised for calendar views and date pickers. +The first day of the week can be customized for calendar views and date pickers. You can choose one of the following options as the first day of the week: @@ -111,3 +127,15 @@ You can choose one of the following options as the first day of the week: - Monday If you select **System Default**, the system-wide default setting will be used. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/autocomplete_characters.md b/doc/user/project/autocomplete_characters.md new file mode 100644 index 00000000000..9ebf7f821a1 --- /dev/null +++ b/doc/user/project/autocomplete_characters.md @@ -0,0 +1,48 @@ +# Autocomplete characters + +The autocomplete characters provide a quick way of entering field values into +Markdown fields. When you start typing a word in a Markdown field with one of +the following characters, GitLab progressively autocompletes against a set of +matching values. The string matching is not case sensitive. + +| Character | Autocompletes | +| :-------- | :------------ | +| `~` | Labels | +| `%` | Milestones | +| `@` | Users and groups | +| `#` | Issues | +| `!` | Merge requests | +| `&` | Epics | +| `$` | Snippets | +| `:` | Emoji | +| `/` | Quick Actions | + +Up to 5 of the most relevant matches are displayed in a popup list. When you +select an item from the list, the value is entered in the field. The more +characters you enter, the more precise the matches are. + +Autocomplete characters are useful when combined with [Quick Actions](quick_actions.md). + +## Example + +Assume your GitLab instance includes the following users: + +| Username | Name | +| :-------------- | :--- | +| alessandra | Rosy Grant | +| lawrence.white | Kelsey Kerluke | +| leanna | Rosemarie Rogahn | +| logan_gutkowski | Lee Wuckert | +| shelba | Josefine Haley | + +In an Issue comment, entering `@l` results in the following popup list +appearing. Note that user `shelba` is not included, because the list includes +only the 5 users most relevant to the Issue. + + + +If you continue to type, `@le`, the popup list changes to the following. The +popup now only includes users where `le` appears in their username, or a word in +their name. + + diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 8849dd2d684..cd3e5f5a63c 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -17,10 +17,10 @@ If you find that you have to add the same badges to several projects, you may wa To add a new badge to a project: -1. Navigate to your project's **Settings > General > Badges**. -1. Under "Link", enter the URL that the badges should point to and under - "Badge image URL" the URL of the image that should be displayed. -1. Submit the badge by clicking the **Add badge** button. +1. Navigate to your project's **Settings > General > Badges**. +1. Under "Link", enter the URL that the badges should point to and under + "Badge image URL" the URL of the image that should be displayed. +1. Submit the badge by clicking the **Add badge** button. After adding a badge to a project, you can see it in the list below the form. You can edit it by clicking on the pen icon next to it or to delete it by @@ -39,10 +39,10 @@ project, consider adding them on the [project level](#project-badges) or use To add a new badge to a group: -1. Navigate to your group's **Settings > General > Badges**. -1. Under "Link", enter the URL that the badges should point to and under - "Badge image URL" the URL of the image that should be displayed. -1. Submit the badge by clicking the **Add badge** button. +1. Navigate to your group's **Settings > General > Badges**. +1. Under "Link", enter the URL that the badges should point to and under + "Badge image URL" the URL of the image that should be displayed. +1. Submit the badge by clicking the **Add badge** button. After adding a badge to a group, you can see it in the list below the form. You can edit the badge by clicking on the pen icon next to it or to delete it diff --git a/doc/user/project/bulk_editing.md b/doc/user/project/bulk_editing.md index d0c7daf4692..f4733620640 100644 --- a/doc/user/project/bulk_editing.md +++ b/doc/user/project/bulk_editing.md @@ -13,13 +13,20 @@ by using the bulk editing feature.  NOTE: **Note:** -Bulk editing of issues and merge requests is only available at the project level. +Bulk editing issues and merge requests is also available at the group level. +For more details, see [bulk editing group issues and merge requests](../group/bulk_editing/index.md). -To update multiple project issues or merge requests at the same time, navigate to -their respective lists and click **Edit issues** or **Edit merge requests** available -in the tab bar. This will open a sidebar on the right-hand side of your screen -where editable fields will be displayed. Checkboxes will also appear to the left-hand -side of eachissue or merge request for you to select the items you want to update. +To update multiple project issues or merge requests at the same time: -Once you have selected all relevant items, choose the appropriate fields and their -values from the sidebar and click **Update all** to apply your changes. +1. Navigate to their respective list. + +1. Click **Edit issues** or **Edit merge requests**. + + - This will open a sidebar on the right-hand side of your screen + where editable fields will be displayed. + + - Checkboxes will also appear beside each issue or merge request. + +1. Check the checkboxes of each items to be edited. +1. Choose the appropriate fields and their values from the sidebar. +1. Click **Update all**. diff --git a/doc/user/project/clusters/eks_and_gitlab/index.md b/doc/user/project/clusters/eks_and_gitlab/index.md index ea09f9f1547..28f3420de35 100644 --- a/doc/user/project/clusters/eks_and_gitlab/index.md +++ b/doc/user/project/clusters/eks_and_gitlab/index.md @@ -40,97 +40,97 @@ then click **Add an existing Kubernetes cluster**. A few details from the EKS cluster will be required to connect it to GitLab: -1. **Retrieve the certificate**: A valid Kubernetes certificate is needed to - authenticate to the EKS cluster. We will use the certificate created by default. - Open a shell and use `kubectl` to retrieve it: +1. **Retrieve the certificate**: A valid Kubernetes certificate is needed to + authenticate to the EKS cluster. We will use the certificate created by default. + Open a shell and use `kubectl` to retrieve it: - - List the secrets with `kubectl get secrets`, and one should named similar to - `default-token-xxxxx`. Copy that token name for use below. - - Get the certificate with: + - List the secrets with `kubectl get secrets`, and one should named similar to + `default-token-xxxxx`. Copy that token name for use below. + - Get the certificate with: - ```sh - kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode - ``` + ```sh + kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode + ``` -1. **Create admin token**: A `cluster-admin` token is required to install and - manage Helm Tiller. GitLab establishes mutual SSL auth with Helm Tiller - and creates limited service accounts for each application. To create the - token we will create an admin service account as follows: +1. **Create admin token**: A `cluster-admin` token is required to install and + manage Helm Tiller. GitLab establishes mutual SSL auth with Helm Tiller + and creates limited service accounts for each application. To create the + token we will create an admin service account as follows: - 2.1. Create a file called `eks-admin-service-account.yaml` with contents: + 2.1. Create a file called `eks-admin-service-account.yaml` with contents: - ```yaml - apiVersion: v1 - kind: ServiceAccount - metadata: - name: eks-admin - namespace: kube-system - ``` + ```yaml + apiVersion: v1 + kind: ServiceAccount + metadata: + name: eks-admin + namespace: kube-system + ``` - 2.2. Apply the service account to your cluster: + 2.2. Apply the service account to your cluster: - ```bash - kubectl apply -f eks-admin-service-account.yaml - ``` + ```bash + kubectl apply -f eks-admin-service-account.yaml + ``` - Output: + Output: - ```bash - serviceaccount "eks-admin" created - ``` + ```bash + serviceaccount "eks-admin" created + ``` - 2.3. Create a file called `eks-admin-cluster-role-binding.yaml` with contents: + 2.3. Create a file called `eks-admin-cluster-role-binding.yaml` with contents: - ```yaml - apiVersion: rbac.authorization.k8s.io/v1beta1 - kind: ClusterRoleBinding - metadata: - name: eks-admin - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: eks-admin - namespace: kube-system - ``` + ```yaml + apiVersion: rbac.authorization.k8s.io/v1beta1 + kind: ClusterRoleBinding + metadata: + name: eks-admin + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: eks-admin + namespace: kube-system + ``` - 2.4. Apply the cluster role binding to your cluster: + 2.4. Apply the cluster role binding to your cluster: - ```bash - kubectl apply -f eks-admin-cluster-role-binding.yaml - ``` + ```bash + kubectl apply -f eks-admin-cluster-role-binding.yaml + ``` - Output: + Output: - ```bash - clusterrolebinding "eks-admin" created - ``` + ```bash + clusterrolebinding "eks-admin" created + ``` - 2.5. Retrieve the token for the `eks-admin` service account: + 2.5. Retrieve the token for the `eks-admin` service account: - ```bash - kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') - ``` + ```bash + kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') + ``` - Copy the `<authentication_token>` value from the output: + Copy the `<authentication_token>` value from the output: - ```yaml - Name: eks-admin-token-b5zv4 - Namespace: kube-system - Labels: <none> - Annotations: kubernetes.io/service-account.name=eks-admin - kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 + ```yaml + Name: eks-admin-token-b5zv4 + Namespace: kube-system + Labels: <none> + Annotations: kubernetes.io/service-account.name=eks-admin + kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 - Type: kubernetes.io/service-account-token + Type: kubernetes.io/service-account-token - Data - ==== - ca.crt: 1025 bytes - namespace: 11 bytes - token: <authentication_token> - ``` + Data + ==== + ca.crt: 1025 bytes + namespace: 11 bytes + token: <authentication_token> + ``` 1. The API server endpoint is also required, so GitLab can connect to the cluster. This is displayed on the AWS EKS console, when viewing the EKS cluster details. @@ -253,7 +253,7 @@ With RBAC disabled and services deployed, [Auto DevOps](../../../../topics/autodevops/index.md) can now be leveraged to build, test, and deploy the app. -[Enable Auto DevOps](../../../../topics/autodevops/index.md#enablingdisabling-auto-devops-at-the-project-level) +[Enable Auto DevOps](../../../../topics/autodevops/index.md#at-the-project-level) if not already enabled. If a wildcard DNS entry was created resolving to the Load Balancer, enter it in the `domain` field under the Auto DevOps settings. Otherwise, the deployed app will not be externally available outside of the cluster. diff --git a/doc/user/project/clusters/img/k8s_cluster_monitoring.png b/doc/user/project/clusters/img/k8s_cluster_monitoring.png Binary files differindex e449893a606..0a8c5043c65 100644 --- a/doc/user/project/clusters/img/k8s_cluster_monitoring.png +++ b/doc/user/project/clusters/img/k8s_cluster_monitoring.png diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 56f8257fbe7..cf3a3fef79f 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -1,33 +1,113 @@ -# Connecting GitLab with a Kubernetes cluster +# Kubernetes clusters -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/35954) in GitLab 10.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/35954) in GitLab 10.1 for projects. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) in +> GitLab 11.6 for [groups](../../group/clusters/index.md). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/39840) in +> GitLab 11.11 for [instances](../../instance/clusters/index.md). -Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes -cluster in a few steps. +GitLab provides many features with a Kubernetes integration. Kubernetes can be +integrated with projects, but also: + +- [Groups](../../group/clusters/index.md). +- [Instances](../../instance/clusters/index.md). NOTE: **Scalable app deployment with GitLab and Google Cloud Platform** [Watch the webcast](https://about.gitlab.com/webcast/scalable-app-deploy/) and learn how to spin up a Kubernetes cluster managed by Google Cloud Platform (GCP) in a few clicks. ## Overview -With one or more Kubernetes clusters associated to your project, you can use -[Review Apps](../../../ci/review_apps/index.md), deploy your applications, run -your pipelines, use it with [Auto DevOps](../../../topics/autodevops/index.md), -and much more, all from within GitLab. +Using the GitLab project Kubernetes integration, you can: + +- Use [Review Apps](../../../ci/review_apps/index.md). +- Run [pipelines](../../../ci/pipelines.md). +- [Deploy](#deploying-to-a-kubernetes-cluster) your applications. +- Detect and [monitor Kubernetes](#kubernetes-monitoring). +- Use it with [Auto DevOps](#auto-devops). +- Use [Web terminals](#web-terminals). +- Use [Deploy Boards](#deploy-boards-premium). **(PREMIUM)** +- Use [Canary Deployments](#canary-deployments-premium). **(PREMIUM)** +- View [Pod logs](#pod-logs-ultimate). **(ULTIMATE)** + +You can also: + +- Connect and deploy to an [Amazon EKS cluster](eks_and_gitlab/index.html). +- Run serverless workloads on [Kubernetes with Knative](serverless/index.md). + +### Deploy Boards **(PREMIUM)** + +GitLab's Deploy Boards offer a consolidated view of the current health and +status of each CI [environment](../../../ci/environments.md) running on Kubernetes, +displaying the status of the pods in the deployment. Developers and other +teammates can view the progress and status of a rollout, pod by pod, in the +workflow they already use without any need to access Kubernetes. + +[Read more about Deploy Boards](../deploy_boards.md) + +### Canary Deployments **(PREMIUM)** + +Leverage [Kubernetes' Canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) +and visualize your canary deployments right inside the Deploy Board, without +the need to leave GitLab. + +[Read more about Canary Deployments](../canary_deployments.md) + +### Pod logs **(ULTIMATE)** + +GitLab makes it easy to view the logs of running pods in connected Kubernetes clusters. By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. + +[Read more about Kubernetes pod logs](kubernetes_pod_logs.md) + +### Kubernetes monitoring + +Automatically detect and monitor Kubernetes metrics. Automatic monitoring of +[NGINX ingress](../integrations/prometheus_library/nginx.md) is also supported. + +[Read more about Kubernetes monitoring](../integrations/prometheus_library/kubernetes.md) + +### Auto DevOps + +Auto DevOps automatically detects, builds, tests, deploys, and monitors your +applications. + +To make full use of Auto DevOps(Auto Deploy, Auto Review Apps, and Auto Monitoring) +you will need the Kubernetes project integration enabled. + +[Read more about Auto DevOps](../../../topics/autodevops/index.md) + +NOTE: **Note** +Kubernetes clusters can be used without Auto DevOps. -There are two options when adding a new cluster to your project; either associate -your account with Google Kubernetes Engine (GKE) so that you can [create new -clusters](#adding-and-creating-a-new-gke-cluster-via-gitlab) from within GitLab, -or provide the credentials to an [existing Kubernetes cluster](#adding-an-existing-kubernetes-cluster). +### Web terminals NOTE: **Note:** -From [GitLab 11.6](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) you -can also associate a Kubernetes cluster to your groups and from -[GitLab 11.11](https://gitlab.com/gitlab-org/gitlab-ce/issues/39840), -to the GitLab instance. Learn more about [group-level](../../group/clusters/index.md) -and [instance-level](../../instance/clusters/index.md) Kubernetes clusters. +Introduced in GitLab 8.15. You must be the project owner or have `maintainer` permissions +to use terminals. Support is limited to the first container in the +first pod of your environment. -## Adding and creating a new GKE cluster via GitLab +When enabled, the Kubernetes service adds [web terminal](../../../ci/environments.md#web-terminals) +support to your [environments](../../../ci/environments.md). This is based on the `exec` functionality found in +Docker and Kubernetes, so you get a new shell session within your existing +containers. To use this integration, you should deploy to Kubernetes using +the deployment variables above, ensuring any deployments, replica sets, and +pods are annotated with: + +- `app.gitlab.com/env: $CI_ENVIRONMENT_SLUG` +- `app.gitlab.com/app: $CI_PROJECT_PATH_SLUG` + +`$CI_ENVIRONMENT_SLUG` and `$CI_PROJECT_PATH_SLUG` are the values of +the CI variables. + +## Adding and removing clusters + +There are two options when adding a new cluster to your project: + +- Associate your account with Google Kubernetes Engine (GKE) to + [create new clusters](#add-new-gke-cluster) from within GitLab. +- Provide credentials to an + [existing Kubernetes cluster](#add-existing-kubernetes-cluster). + +### Add new GKE cluster TIP: **Tip:** Every new Google Cloud Platform (GCP) account receives [$300 in credit upon sign up](https://console.cloud.google.com/freetrial), @@ -39,7 +119,7 @@ The [Google authentication integration](../../../integration/google.md) must be enabled in GitLab at the instance level. If that's not the case, ask your GitLab administrator to enable it. On GitLab.com, this is enabled. -### Requirements +#### Requirements Before creating your first cluster on Google Kubernetes Engine with GitLab's integration, make sure the following requirements are met: @@ -49,15 +129,15 @@ integration, make sure the following requirements are met: - The Kubernetes Engine API and related service are enabled. It should work immediately but may take up to 10 minutes after you create a project. For more information see the ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin). -### Creating the cluster +#### Creating the cluster If all of the above requirements are met, you can proceed to create and add a new Kubernetes cluster to your project: 1. Navigate to your project's **Operations > Kubernetes** page. - NOTE: **Note:** - You need Maintainer [permissions] and above to access the Kubernetes page. + NOTE: **Note:** + You need Maintainer [permissions](../../permissions.md) and above to access the Kubernetes page. 1. Click **Add Kubernetes cluster**. 1. Click **Create with Google Kubernetes Engine**. @@ -91,130 +171,164 @@ client certificate is enabled. NOTE: **Note:** Starting from [GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ce/issues/55902), all GKE clusters created by GitLab are RBAC enabled. Take a look at the [RBAC section](#rbac-cluster-resources) for more information. -## Adding an existing Kubernetes cluster +### Add existing Kubernetes cluster + +NOTE: **Note:** +Kubernetes integration is not supported for arm64 clusters. See the issue [Helm Tiller fails to install on arm64 cluster](https://gitlab.com/gitlab-org/gitlab-ce/issues/64044) for details. To add an existing Kubernetes cluster to your project: 1. Navigate to your project's **Operations > Kubernetes** page. - NOTE: **Note:** - You need Maintainer [permissions] and above to access the Kubernetes page. + NOTE: **Note:** + You need Maintainer [permissions](../../permissions.md) and above to access the Kubernetes page. 1. Click **Add Kubernetes cluster**. 1. Click **Add an existing Kubernetes cluster** and fill in the details: - - **Kubernetes cluster name** (required) - The name you wish to give the cluster. - - **Environment scope** (required) - The - [associated environment](#setting-the-environment-scope-premium) to this cluster. - - **API URL** (required) - - It's the URL that GitLab uses to access the Kubernetes API. Kubernetes - exposes several APIs, we want the "base" URL that is common to all of them, - e.g., `https://kubernetes.example.com` rather than `https://kubernetes.example.com/api/v1`. - - Get the API URL by running this command: - - ```sh - kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}' - ``` - - **CA certificate** (required) - A valid Kubernetes certificate is needed to authenticate to the EKS cluster. We will use the certificate created by default. - - List the secrets with `kubectl get secrets`, and one should named similar to - `default-token-xxxxx`. Copy that token name for use below. - - Get the certificate by running this command: - - ```sh - kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode - ``` - - **Token** - - GitLab authenticates against Kubernetes using service tokens, which are - scoped to a particular `namespace`. - **The token used should belong to a service account with - [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) - privileges.** To create this service account: - - 1. Create a file called `gitlab-admin-service-account.yaml` with contents: - - ```yaml - apiVersion: v1 - kind: ServiceAccount - metadata: - name: gitlab-admin - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1beta1 - kind: ClusterRoleBinding - metadata: - name: gitlab-admin - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: gitlab-admin - namespace: kube-system - ``` - - 1. Apply the service account and cluster role binding to your cluster: - - ```bash - kubectl apply -f gitlab-admin-service-account.yaml - ``` - - Output: - - ```bash - serviceaccount "gitlab-admin" created - clusterrolebinding "gitlab-admin" created - ``` - - 1. Retrieve the token for the `gitlab-admin` service account: - - ```bash - kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}') - ``` - - Copy the `<authentication_token>` value from the output: - - ```yaml - Name: gitlab-admin-token-b5zv4 - Namespace: kube-system - Labels: <none> - Annotations: kubernetes.io/service-account.name=gitlab-admin - kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 - - Type: kubernetes.io/service-account-token - - Data - ==== - ca.crt: 1025 bytes - namespace: 11 bytes - token: <authentication_token> - ``` - - NOTE: **Note:** - For GKE clusters, you will need the - `container.clusterRoleBindings.create` permission to create a cluster - role binding. You can follow the [Google Cloud - documentation](https://cloud.google.com/iam/docs/granting-changing-revoking-access) - to grant access. - - - **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. See the [Managed clusters section](#gitlab-managed-clusters) for more information. - - - **Project namespace** (optional) - You don't have to fill it in; by leaving - it blank, GitLab will create one for you. Also: - - Each project should have a unique namespace. - - The project namespace is not necessarily the namespace of the secret, if - you're using a secret with broader permissions, like the secret from `default`. - - You should **not** use `default` as the project namespace. - - If you or someone created a secret specifically for the project, usually - with limited permissions, the secret's namespace and project namespace may - be the same. + - **Kubernetes cluster name** (required) - The name you wish to give the cluster. + - **Environment scope** (required) - The + [associated environment](#setting-the-environment-scope-premium) to this cluster. + - **API URL** (required) - + It's the URL that GitLab uses to access the Kubernetes API. Kubernetes + exposes several APIs, we want the "base" URL that is common to all of them, + e.g., `https://kubernetes.example.com` rather than `https://kubernetes.example.com/api/v1`. + + Get the API URL by running this command: + + ```sh + kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}' + ``` + + - **CA certificate** (required) - A valid Kubernetes certificate is needed to authenticate to the EKS cluster. We will use the certificate created by default. + - List the secrets with `kubectl get secrets`, and one should named similar to + `default-token-xxxxx`. Copy that token name for use below. + - Get the certificate by running this command: + + ```sh + kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode + ``` + + - **Token** - + GitLab authenticates against Kubernetes using service tokens, which are + scoped to a particular `namespace`. + **The token used should belong to a service account with + [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) + privileges.** To create this service account: + + 1. Create a file called `gitlab-admin-service-account.yaml` with contents: + + ```yaml + apiVersion: v1 + kind: ServiceAccount + metadata: + name: gitlab-admin + namespace: kube-system + --- + apiVersion: rbac.authorization.k8s.io/v1beta1 + kind: ClusterRoleBinding + metadata: + name: gitlab-admin + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: gitlab-admin + namespace: kube-system + ``` + + 1. Apply the service account and cluster role binding to your cluster: + + ```bash + kubectl apply -f gitlab-admin-service-account.yaml + ``` + + Output: + + ```bash + serviceaccount "gitlab-admin" created + clusterrolebinding "gitlab-admin" created + ``` + + 1. Retrieve the token for the `gitlab-admin` service account: + + ```bash + kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}') + ``` + + Copy the `<authentication_token>` value from the output: + + ```yaml + Name: gitlab-admin-token-b5zv4 + Namespace: kube-system + Labels: <none> + Annotations: kubernetes.io/service-account.name=gitlab-admin + kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 + + Type: kubernetes.io/service-account-token + + Data + ==== + ca.crt: 1025 bytes + namespace: 11 bytes + token: <authentication_token> + ``` + + NOTE: **Note:** + For GKE clusters, you will need the + `container.clusterRoleBindings.create` permission to create a cluster + role binding. You can follow the [Google Cloud + documentation](https://cloud.google.com/iam/docs/granting-changing-revoking-access) + to grant access. + + - **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. See the [Managed clusters section](#gitlab-managed-clusters) for more information. + + - **Project namespace** (optional) - You don't have to fill it in; by leaving + it blank, GitLab will create one for you. Also: + - Each project should have a unique namespace. + - The project namespace is not necessarily the namespace of the secret, if + you're using a secret with broader permissions, like the secret from `default`. + - You should **not** use `default` as the project namespace. + - If you or someone created a secret specifically for the project, usually + with limited permissions, the secret's namespace and project namespace may + be the same. 1. Finally, click the **Create Kubernetes cluster** button. After a couple of minutes, your cluster will be ready to go. You can now proceed to install some [pre-defined applications](#installing-applications). -## Security implications +### Enabling or disabling integration + +After you have successfully added your cluster information, you can enable the +Kubernetes cluster integration: + +1. Click the **Enabled/Disabled** switch +1. Hit **Save** for the changes to take effect + +To disable the Kubernetes cluster integration, follow the same procedure. + +### Removing integration + +NOTE: **Note:** +You need Maintainer [permissions](../../permissions.md) and above to remove a Kubernetes cluster integration. + +NOTE: **Note:** +When you remove a cluster, you only remove its relation to GitLab, not the +cluster itself. To remove the cluster, you can do so by visiting the GKE +dashboard or using `kubectl`. + +To remove the Kubernetes cluster integration from your project, simply click the +**Remove integration** button. You will then be able to follow the procedure +and add a Kubernetes cluster again. + +## Cluster configuration + +This section covers important considerations for configuring Kubernetes +clusters with GitLab. + +### Security implications CAUTION: **Important:** The whole cluster security is based on a model where [developers](../../permissions.md) @@ -225,7 +339,7 @@ functionalities needed to successfully build and deploy a containerized application. Bear in mind that the same credentials are used for all the applications running on the cluster. -## GitLab-managed clusters +### GitLab-managed clusters > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22011) in GitLab 11.5. > Became [optional](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26565) in GitLab 11.11. @@ -244,7 +358,7 @@ NOTE: **Note:** If you [install applications](#installing-applications) on your cluster, GitLab will create the resources required to run these even if you have chosen to manage your own cluster. -## Base domain +### Base domain > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24580) in GitLab 11.8. @@ -262,7 +376,7 @@ you can either: - Create an `A` record that points to the Ingress IP address with your domain provider. - Enter a wildcard DNS address using a service such as nip.io or xip.io. For example, `192.168.1.1.xip.io`. -## Access controls +### Access controls When creating a cluster in GitLab, you will be asked if you would like to create either: @@ -273,13 +387,9 @@ NOTE: **Note:** [RBAC](#rbac-cluster-resources) is recommended and the GitLab default. GitLab creates the necessary service accounts and privileges to install and run -[GitLab managed applications](#installing-applications). When GitLab creates the cluster: - -- A `gitlab` service account with `cluster-admin` privileges is created in the `default` namespace - to manage the newly created cluster. -- A project service account with [`edit` - privileges](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) - is created in the GitLab-created project namespace for [deployment jobs](#deployment-variables). +[GitLab managed applications](#installing-applications). When GitLab creates the cluster, +a `gitlab` service account with `cluster-admin` privileges is created in the `default` namespace +to manage the newly created cluster. NOTE: **Note:** Restricted service account for deployment was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/51716) in GitLab 11.5. @@ -292,47 +402,53 @@ Helm will also create additional service accounts and other resources for each installed application. Consult the documentation of the Helm charts for each application for details. -If you are [adding an existing Kubernetes cluster](#adding-an-existing-kubernetes-cluster), +If you are [adding an existing Kubernetes cluster](#add-existing-kubernetes-cluster), ensure the token of the account has administrator privileges for the cluster. The resources created by GitLab differ depending on the type of cluster. -### ABAC cluster resources +#### ABAC cluster resources GitLab creates the following resources for ABAC clusters. -| Name | Type | Details | Created when | -|:------------------|:---------------------|:----------------------------------|:---------------------------| -| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | -| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | -| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | -| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | -| Project namespace | `ServiceAccount` | Uses namespace of Project | Deploying to a cluster | -| Project namespace | `Secret` | Token for project ServiceAccount | Deploying to a cluster | +| Name | Type | Details | Created when | +|:----------------------|:---------------------|:-------------------------------------|:---------------------------| +| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | +| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | +| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | +| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | +| Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | +| Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | +| Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | -### RBAC cluster resources +#### RBAC cluster resources GitLab creates the following resources for RBAC clusters. -| Name | Type | Details | Created when | -|:------------------|:---------------------|:-----------------------------------------------------------------------------------------------------------|:---------------------------| -| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | -| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new GKE Cluster | -| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | -| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | -| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | -| Project namespace | `ServiceAccount` | Uses namespace of Project | Deploying to a cluster | -| Project namespace | `Secret` | Token for project ServiceAccount | Deploying to a cluster | -| Project namespace | `RoleBinding` | [`edit`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Deploying to a cluster | +| Name | Type | Details | Created when | +|:----------------------|:---------------------|:-----------------------------------------------------------------------------------------------------------|:---------------------------| +| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | +| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new GKE Cluster | +| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | +| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | +| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | +| Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | +| Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | +| Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | +| Environment namespace | `RoleBinding` | [`edit`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Deploying to a cluster | + +NOTE: **Note:** +Environment-specific resources are only created if your cluster is [managed by GitLab](#gitlab-managed-clusters). NOTE: **Note:** -Project-specific resources are only created if your cluster is [managed by GitLab](#gitlab-managed-clusters). +If your cluster was created before GitLab 12.2, it will use a single namespace for all project environments. -### Security of GitLab Runners +#### Security of GitLab Runners GitLab Runners have the [privileged mode](https://docs.gitlab.com/runner/executors/docker.html#the-privileged-mode) enabled by default, which allows them to execute special commands and running -Docker in Docker. This functionality is needed to run some of the [Auto DevOps] +Docker in Docker. This functionality is needed to run some of the +[Auto DevOps](../../../topics/autodevops/index.md) jobs. This implies the containers are running in privileged mode and you should, therefore, be aware of some important details. @@ -341,10 +457,81 @@ turn can do almost everything that the host can do. Be aware of the inherent security risk associated with performing `docker run` operations on arbitrary images as they effectively have root access. -If you don't want to use GitLab Runner in privileged mode, first make sure that -you don't have it installed via the applications, and then use the -[Runner's Helm chart](../../../install/kubernetes/gitlab_runner_chart.md) to -install it manually. +If you don't want to use GitLab Runner in privileged mode, either: + +- Use shared Runners on GitLab.com. They don't have this security issue. +- Set up your own Runners using configuration described at + [Shared Runners](../../gitlab_com/index.md#shared-runners). This involves: + 1. Making sure that you don't have it installed via + [the applications](#installing-applications). + 1. Installing a Runner + [using `docker+machine`](https://docs.gitlab.com/runner/executors/docker_machine.html). + +### Setting the environment scope **(PREMIUM)** + +When adding more than one Kubernetes cluster to your project, you need to differentiate +them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the +[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables) work. + +The default environment scope is `*`, which means all jobs, regardless of their +environment, will use that cluster. Each scope can only be used by a single +cluster in a project, and a validation error will occur if otherwise. +Also, jobs that don't have an environment keyword set will not be able to access any cluster. + +For example, let's say the following Kubernetes clusters exist in a project: + +| Cluster | Environment scope | +| ----------- | ----------------- | +| Development | `*` | +| Production | `production` | + +And the following environments are set in +[`.gitlab-ci.yml`](../../../ci/yaml/README.md): + +```yaml +stages: +- test +- deploy + +test: + stage: test + script: sh test + +deploy to staging: + stage: deploy + script: make deploy + environment: + name: staging + url: https://staging.example.com/ + +deploy to production: + stage: deploy + script: make deploy + environment: + name: production + url: https://example.com/ +``` + +The result will then be: + +- The Development cluster details will be available in the `deploy to staging` + job. +- The production cluster details will be available in the `deploy to production` + job. +- No cluster details will be available in the `test` job because it doesn't + define any environment. + +### Multiple Kubernetes clusters **(PREMIUM)** + +> Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. + +With GitLab Premium, you can associate more than one Kubernetes cluster to your +project. That way you can have different clusters for different environments, +like dev, staging, production, etc. + +Simply add another cluster, like you did the first time, and make sure to +[set an environment scope](#setting-the-environment-scope-premium) that will +differentiate the new cluster with the rest. ## Installing applications @@ -353,7 +540,7 @@ cluster. For more information on installing, upgrading, uninstalling, and troubleshooting applications for your project cluster, see [Gitlab Managed Apps](../../clusters/applications.md). -## Getting the external endpoint +### Getting the external endpoint NOTE: **Note:** With the following procedure, a load balancer must be installed in your cluster @@ -364,7 +551,7 @@ to obtain the endpoint. You can use either In order to publish your web application, you first need to find the endpoint which will be either an IP address or a hostname associated with your load balancer. -### Automatically determining the external endpoint +#### Automatically determining the external endpoint > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17052) in GitLab 10.6. @@ -379,7 +566,7 @@ and your cluster runs on Google Kubernetes Engine: If GitLab is still unable to determine the endpoint of your Ingress or Knative application, you can manually determine it by following the steps below. -### Manually determining the external endpoint +#### Manually determining the external endpoint If the cluster is on GKE, click the **Google Kubernetes Engine** link in the **Advanced settings**, or go directly to the @@ -418,7 +605,7 @@ Otherwise, you can list the IP addresses of all load balancers: kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} ' ``` -### Using a static IP +#### Using a static IP By default, an ephemeral external IP address is associated to the cluster's load balancer. If you associate the ephemeral IP with your DNS and the IP changes, @@ -428,79 +615,27 @@ reserved IP. Read how to [promote an ephemeral external IP address in GKE](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip). -### Pointing your DNS at the external endpoint +#### Pointing your DNS at the external endpoint Once you've set up the external endpoint, you should associate it with a [wildcard DNS record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) such as `*.example.com.` in order to be able to reach your apps. If your external endpoint is an IP address, use an A record. If your external endpoint is a hostname, use a CNAME record. -## Multiple Kubernetes clusters **(PREMIUM)** - -> Introduced in [GitLab Premium][ee] 10.3. - -With GitLab Premium, you can associate more than one Kubernetes clusters to your -project. That way you can have different clusters for different environments, -like dev, staging, production, etc. - -Simply add another cluster, like you did the first time, and make sure to -[set an environment scope](#setting-the-environment-scope-premium) that will -differentiate the new cluster with the rest. - -## Setting the environment scope **(PREMIUM)** - -When adding more than one Kubernetes cluster to your project, you need to differentiate -them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the -[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium) work. - -The default environment scope is `*`, which means all jobs, regardless of their -environment, will use that cluster. Each scope can only be used by a single -cluster in a project, and a validation error will occur if otherwise. -Also, jobs that don't have an environment keyword set will not be able to access any cluster. - ---- - -For example, let's say the following Kubernetes clusters exist in a project: - -| Cluster | Environment scope | -| ----------- | ----------------- | -| Development | `*` | -| Staging | `staging` | -| Production | `production` | - -And the following environments are set in [`.gitlab-ci.yml`](../../../ci/yaml/README.md): - -```yaml -stages: -- test -- deploy - -test: - stage: test - script: sh test +## Deploying to a Kubernetes cluster -deploy to staging: - stage: deploy - script: make deploy - environment: - name: staging - url: https://staging.example.com/ +A Kubernetes cluster can be the destination for a deployment job. If -deploy to production: - stage: deploy - script: make deploy - environment: - name: production - url: https://example.com/ -``` +- The cluster is integrated with GitLab, special + [deployment variables](#deployment-variables) are made available to your job + and configuration is not required. You can immediately begin interacting with + the cluster from your jobs using tools such as `kubectl` or `helm`. +- You don't use GitLab's cluster integration you can still deploy to your + cluster. However, you will need configure Kubernetes tools yourself + using [environment variables](../../../ci/variables/README.md#creating-a-custom-environment-variable) + before you can interact with the cluster from your jobs. -The result will then be: - -- The development cluster will be used for the "test" job. -- The staging cluster will be used for the "deploy to staging" job. -- The production cluster will be used for the "deploy to production" job. - -## Deployment variables +### Deployment variables The Kubernetes cluster integration exposes the following [deployment variables](../../../ci/variables/README.md#deployment-environment-variables) in the @@ -509,8 +644,8 @@ GitLab CI/CD build environment. | Variable | Description | | -------- | ----------- | | `KUBE_URL` | Equal to the API URL. | -| `KUBE_TOKEN` | The Kubernetes token of the [project service account](#access-controls). | -| `KUBE_NAMESPACE` | The Kubernetes namespace is auto-generated if not specified. The default value is `<project_name>-<project_id>`. You can overwrite it to use different one if needed, otherwise the `KUBE_NAMESPACE` variable will receive the default value. | +| `KUBE_TOKEN` | The Kubernetes token of the [environment service account](#access-controls). | +| `KUBE_NAMESPACE` | The Kubernetes namespace is auto-generated if not specified. The default value is `<project_name>-<project_id>-<environment>`. You can overwrite it to use different one if needed, otherwise the `KUBE_NAMESPACE` variable will receive the default value. | | `KUBE_CA_PEM_FILE` | Path to a file containing PEM data. Only present if a custom CA bundle was specified. | | `KUBE_CA_PEM` | (**deprecated**) Raw PEM data. Only if a custom CA bundle was specified. | | `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. This config also embeds the same token defined in `KUBE_TOKEN` so you likely will only need this variable. This variable name is also automatically picked up by `kubectl` so you won't actually need to reference it explicitly if using `kubectl`. | @@ -520,7 +655,10 @@ NOTE: **NOTE:** Prior to GitLab 11.5, `KUBE_TOKEN` was the Kubernetes token of the main service account of the cluster integration. -### Troubleshooting failed deployment jobs +NOTE: **Note:** +If your cluster was created before GitLab 12.2, default `KUBE_NAMESPACE` will be set to `<project_name>-<project_id>`. + +### Troubleshooting Before the deployment jobs starts, GitLab creates the following specifically for the deployment job: @@ -552,105 +690,8 @@ namespaces and service accounts yourself. ## Monitoring your Kubernetes cluster **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4701) in [GitLab Ultimate][ee] 10.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4701) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. When [Prometheus is deployed](#installing-applications), GitLab will automatically monitor the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start.  - -## Enabling or disabling the Kubernetes cluster integration - -After you have successfully added your cluster information, you can enable the -Kubernetes cluster integration: - -1. Click the **Enabled/Disabled** switch -1. Hit **Save** for the changes to take effect - -You can now start using your Kubernetes cluster for your deployments. - -To disable the Kubernetes cluster integration, follow the same procedure. - -## Removing the Kubernetes cluster integration - -NOTE: **Note:** -You need Maintainer [permissions] and above to remove a Kubernetes cluster integration. - -NOTE: **Note:** -When you remove a cluster, you only remove its relation to GitLab, not the -cluster itself. To remove the cluster, you can do so by visiting the GKE -dashboard or using `kubectl`. - -To remove the Kubernetes cluster integration from your project, simply click the -**Remove integration** button. You will then be able to follow the procedure -and add a Kubernetes cluster again. - -## What you can get with the Kubernetes integration - -Here's what you can do with GitLab if you enable the Kubernetes integration. - -### Deploy Boards **(PREMIUM)** - -GitLab's Deploy Boards offer a consolidated view of the current health and -status of each CI [environment](../../../ci/environments.md) running on Kubernetes, -displaying the status of the pods in the deployment. Developers and other -teammates can view the progress and status of a rollout, pod by pod, in the -workflow they already use without any need to access Kubernetes. - -[Read more about Deploy Boards](../deploy_boards.md) - -### Canary Deployments **(PREMIUM)** - -Leverage [Kubernetes' Canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) -and visualize your canary deployments right inside the Deploy Board, without -the need to leave GitLab. - -[Read more about Canary Deployments](../canary_deployments.md) - -### Pod logs **(ULTIMATE)** - -GitLab makes it easy to view the logs of running pods in connected Kubernetes clusters. By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. - -[Read more about Kubernetes pod logs](kubernetes_pod_logs.md) - -### Kubernetes monitoring - -Automatically detect and monitor Kubernetes metrics. Automatic monitoring of -[NGINX ingress](../integrations/prometheus_library/nginx.md) is also supported. - -[Read more about Kubernetes monitoring](../integrations/prometheus_library/kubernetes.md) - -### Auto DevOps - -Auto DevOps automatically detects, builds, tests, deploys, and monitors your -applications. - -To make full use of Auto DevOps(Auto Deploy, Auto Review Apps, and Auto Monitoring) -you will need the Kubernetes project integration enabled. - -[Read more about Auto DevOps](../../../topics/autodevops/index.md) - -### Web terminals - -NOTE: **Note:** -Introduced in GitLab 8.15. You must be the project owner or have `maintainer` permissions -to use terminals. Support is limited to the first container in the -first pod of your environment. - -When enabled, the Kubernetes service adds [web terminal](../../../ci/environments.md#web-terminals) -support to your [environments](../../../ci/environments.md). This is based on the `exec` functionality found in -Docker and Kubernetes, so you get a new shell session within your existing -containers. To use this integration, you should deploy to Kubernetes using -the deployment variables above, ensuring any pods you create are labelled with -`app=$CI_ENVIRONMENT_SLUG`. GitLab will do the rest! - -### Integrating Amazon EKS cluster with GitLab - -- Learn how to [connect and deploy to an Amazon EKS cluster](eks_and_gitlab/index.md). - -### Serverless - -- [Run serverless workloads on Kubernetes with Knative.](serverless/index.md) - -[permissions]: ../../permissions.md -[ee]: https://about.gitlab.com/pricing/ -[Auto DevOps]: ../../../topics/autodevops/index.md diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index 864cd75823c..163f3befeee 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -5,6 +5,10 @@ GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. +NOTE: **Kubernetes + GitLab** +Everything you need to build, test, deploy, and run your app at scale. +[Learn more](https://about.gitlab.com/solutions/kubernetes/). + ## Overview [Kubernetes](https://kubernetes.io) pod logs can be viewed directly within GitLab. Logs can be displayed by clicking on a specific pod from [Deploy Boards](../deploy_boards.md): diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index c67b12fb91a..8df27976662 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -1,28 +1,27 @@ # Runbooks -Runbooks are a collection of documented procedures that explain how to -carry out a particular process, be it starting, stopping, debugging, +Runbooks are a collection of documented procedures that explain how to +carry out a particular process, be it starting, stopping, debugging, or troubleshooting a particular system. Using [Jupyter Notebooks](https://jupyter.org/) and the [Rubix library](https://github.com/Nurtch/rubix), users can get started writing their own executable runbooks. - ## Overview -Historically, runbooks took the form of a decision tree or a detailed -step-by-step guide depending on the condition or system. +Historically, runbooks took the form of a decision tree or a detailed +step-by-step guide depending on the condition or system. -Modern implementations have introduced the concept of an "executable -runbooks", where, along with a well-defined process, operators can execute +Modern implementations have introduced the concept of an "executable +runbooks", where, along with a well-defined process, operators can execute pre-written code blocks or database queries against a given environment. ## Executable Runbooks > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45912) in GitLab 11.4. -The JupyterHub app offered via GitLab’s Kubernetes integration now ships -with Nurtch’s Rubix library, providing a simple way to create DevOps +The JupyterHub app offered via GitLab’s Kubernetes integration now ships +with Nurtch’s Rubix library, providing a simple way to create DevOps runbooks. A sample runbook is provided, showcasing common operations. While Rubix makes it simple to create common Kubernetes and AWS workflows, you can also create them manually without Rubix. @@ -35,33 +34,33 @@ for an overview of how this is accomplished in GitLab!** To create an executable runbook, you will need: -1. **Kubernetes** - A Kubernetes cluster is required to deploy the rest of the applications. - The simplest way to get started is to add a cluster using [GitLab's GKE integration](../index.md#adding-and-creating-a-new-gke-cluster-via-gitlab). -1. **Helm Tiller** - Helm is a package manager for Kubernetes and is required to install - all the other applications. It is installed in its own pod inside the cluster which +1. **Kubernetes** - A Kubernetes cluster is required to deploy the rest of the applications. + The simplest way to get started is to add a cluster using [GitLab's GKE integration](../index.md#add-new-gke-cluster). +1. **Helm Tiller** - Helm is a package manager for Kubernetes and is required to install + all the other applications. It is installed in its own pod inside the cluster which can run the helm CLI in a safe environment. -1. **Ingress** - Ingress can provide load balancing, SSL termination, and name-based +1. **Ingress** - Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications. -1. **JupyterHub** - [JupyterHub](https://jupyterhub.readthedocs.io/) is a multi-user service for managing notebooks across - a team. Jupyter Notebooks provide a web-based interactive programming environment +1. **JupyterHub** - [JupyterHub](https://jupyterhub.readthedocs.io/) is a multi-user service for managing notebooks across + a team. Jupyter Notebooks provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. ## Nurtch -Nurtch is the company behind the [Rubix library](https://github.com/Nurtch/rubix). Rubix is -an open-source python library that makes it easy to perform common DevOps tasks inside Jupyter Notebooks. -Tasks such as plotting Cloudwatch metrics and rolling your ECS/Kubernetes app are simplified -down to a couple of lines of code. See the [Nurtch Documentation](http://docs.nurtch.com/en/latest) +Nurtch is the company behind the [Rubix library](https://github.com/Nurtch/rubix). Rubix is +an open-source python library that makes it easy to perform common DevOps tasks inside Jupyter Notebooks. +Tasks such as plotting Cloudwatch metrics and rolling your ECS/Kubernetes app are simplified +down to a couple of lines of code. See the [Nurtch Documentation](http://docs.nurtch.com/en/latest) for more information. ## Configure an executable runbook with GitLab -Follow this step-by-step guide to configure an executable runbook in GitLab using +Follow this step-by-step guide to configure an executable runbook in GitLab using the components outlined above and the preloaded demo runbook. ### 1. Add a Kubernetes cluster -Follow the steps outlined in [Adding and creating a new GKE cluster via GitLab](../index.md#adding-and-creating-a-new-gke-cluster-via-gitlab) +Follow the steps outlined in [Add new GKE cluster](../index.md#add-new-gke-cluster) to add a Kubernetes cluster to your project. ### 2. Install Helm Tiller, Ingress, and JupyterHub @@ -80,13 +79,13 @@ Once Ingress has been installed successfully, click the **Install** button next ### 3. Login to JupyterHub and start the server -Once JupyterHub has been installed successfully, navigate to the displayed **Jupyter Hostname** URL and click -**Sign in with GitLab**. Authentication is automatically enabled for any user of the GitLab instance via OAuth2. This +Once JupyterHub has been installed successfully, navigate to the displayed **Jupyter Hostname** URL and click +**Sign in with GitLab**. Authentication is automatically enabled for any user of the GitLab instance via OAuth2. This will redirect to GitLab in order to authorize JupyterHub to use your GitLab account. Click **Authorize**.  -Once the application has been authorized you will taken back to the JupyterHub application. Click **Start My Server**. +Once the application has been authorized you will taken back to the JupyterHub application. Click **Start My Server**. The server will take a couple of seconds to start. ### 4. Configure access @@ -103,7 +102,7 @@ Double-click the "Nurtch-DevOps-Demo.ipynb" runbook.  -The contents on the runbook will be displayed on the right side of the screen. Under the "Setup" section, you will find +The contents on the runbook will be displayed on the right side of the screen. Under the "Setup" section, you will find entries for both your `PRIVATE_TOKEN` and your `PROJECT_ID`. Enter both these values, conserving the single quotes as follows: ```sql @@ -111,7 +110,7 @@ PRIVATE_TOKEN = 'n671WNGecHugsdEDPsyo' PROJECT_ID = '1234567' ``` -Update the `VARIABLE_NAME` on the last line of this section to match the name of the variable you are using for your +Update the `VARIABLE_NAME` on the last line of this section to match the name of the variable you are using for your access token. In this example our variable name is `PRIVATE_TOKEN`. ```sql @@ -120,8 +119,8 @@ VARIABLE_VALUE = project.variables.get('PRIVATE_TOKEN').value ### 5. Configure an operation -For this example we'll use the "**Run SQL queries in Notebook**" section in the sample runbook to query -a postgres database. The first 4 lines of the section define the variables that are required for this query to function. +For this example we'll use the "**Run SQL queries in Notebook**" section in the sample runbook to query +a postgres database. The first 4 lines of the section define the variables that are required for this query to function. ```sql %env DB_USER={project.variables.get('DB_USER').value} @@ -134,10 +133,10 @@ Create the matching variables in your project's **Settings >> CI/CD >> Variables  -Back in Jupyter, click the "Run SQL queries in Notebook" heading and the click *Run*. The results will be +Back in Jupyter, click the "Run SQL queries in Notebook" heading and the click *Run*. The results will be displayed in-line as follows:  -You can try other operations such as running shell scripts or interacting with a Kubernetes cluster. Visit the -[Nurtch Documentation](http://docs.nurtch.com/) for more information.
\ No newline at end of file +You can try other operations such as running shell scripts or interacting with a Kubernetes cluster. Visit the +[Nurtch Documentation](http://docs.nurtch.com/) for more information. diff --git a/doc/user/project/clusters/serverless/img/dns-entry.png b/doc/user/project/clusters/serverless/img/dns-entry.png Binary files differindex 351e40b77d5..7b5d6497f0e 100644 --- a/doc/user/project/clusters/serverless/img/dns-entry.png +++ b/doc/user/project/clusters/serverless/img/dns-entry.png diff --git a/doc/user/project/clusters/serverless/img/function-details-loaded.png b/doc/user/project/clusters/serverless/img/function-details-loaded.png Binary files differindex 34465c5c087..2f0d61f8032 100644 --- a/doc/user/project/clusters/serverless/img/function-details-loaded.png +++ b/doc/user/project/clusters/serverless/img/function-details-loaded.png diff --git a/doc/user/project/clusters/serverless/img/function-endpoint.png b/doc/user/project/clusters/serverless/img/function-endpoint.png Binary files differindex f3c7ae7a00d..a38fe2cb6c2 100644 --- a/doc/user/project/clusters/serverless/img/function-endpoint.png +++ b/doc/user/project/clusters/serverless/img/function-endpoint.png diff --git a/doc/user/project/clusters/serverless/img/function-execution.png b/doc/user/project/clusters/serverless/img/function-execution.png Binary files differindex 93b0b6d802d..f60dd277081 100644 --- a/doc/user/project/clusters/serverless/img/function-execution.png +++ b/doc/user/project/clusters/serverless/img/function-execution.png diff --git a/doc/user/project/clusters/serverless/img/install-knative.png b/doc/user/project/clusters/serverless/img/install-knative.png Binary files differindex ecc2f8fb481..1dc830848f2 100644 --- a/doc/user/project/clusters/serverless/img/install-knative.png +++ b/doc/user/project/clusters/serverless/img/install-knative.png diff --git a/doc/user/project/clusters/serverless/img/serverless-page.png b/doc/user/project/clusters/serverless/img/serverless-page.png Binary files differindex a872fda7740..8dce3cb1f70 100644 --- a/doc/user/project/clusters/serverless/img/serverless-page.png +++ b/doc/user/project/clusters/serverless/img/serverless-page.png diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index a06c3d3c662..bcf9a677a40 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -24,34 +24,33 @@ To run Knative on Gitlab, you will need: 1. **Existing GitLab project:** You will need a GitLab project to associate all resources. The simplest way to get started: - - If you are planning on deploying functions, clone the [functions example project](https://gitlab.com/knative-examples/functions) to get started. - - If you are planning on deploying a serverless application, clone the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. + - If you are planning on deploying functions, clone the [functions example project](https://gitlab.com/knative-examples/functions) to get started. + - If you are planning on deploying a serverless application, clone the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. 1. **Kubernetes Cluster:** An RBAC-enabled Kubernetes cluster is required to deploy Knative. - The simplest way to get started is to add a cluster using [GitLab's GKE integration](../index.md#adding-and-creating-a-new-gke-cluster-via-gitlab). - The set of minimum recommended cluster specifications to run Knative is 3 nodes, 6 vCPUs, and 22.50 GB memory. + The simplest way to get started is to add a cluster using [GitLab's GKE integration](../index.md#add-new-gke-cluster). + The set of minimum recommended cluster specifications to run Knative is 3 nodes, 6 vCPUs, and 22.50 GB memory. 1. **Helm Tiller:** Helm is a package manager for Kubernetes and is required to install - Knative. + Knative. 1. **GitLab Runner:** A runner is required to run the CI jobs that will deploy serverless - applications or functions onto your cluster. You can install the GitLab Runner - onto the existing Kubernetes cluster. See [Installing Applications](../index.md#installing-applications) for more information. + applications or functions onto your cluster. You can install the GitLab Runner + onto the existing Kubernetes cluster. See [Installing Applications](../index.md#installing-applications) for more information. 1. **Domain Name:** Knative will provide its own load balancer using Istio. It will provide an - external IP address or hostname for all the applications served by Knative. You will be prompted to enter a - wildcard domain where your applications will be served. Configure your DNS server to use the - external IP address or hostname for that domain. + external IP address or hostname for all the applications served by Knative. You will be prompted to enter a + wildcard domain where your applications will be served. Configure your DNS server to use the + external IP address or hostname for that domain. 1. **`.gitlab-ci.yml`:** GitLab uses [Kaniko](https://github.com/GoogleContainerTools/kaniko) - to build the application. We also use [gitlabktl](https://gitlab.com/gitlab-org/gitlabktl) - and [TriggerMesh CLI](https://github.com/triggermesh/tm) CLIs to simplify the - deployment of services and functions to Knative. + to build the application. We also use [gitlabktl](https://gitlab.com/gitlab-org/gitlabktl) + CLI to simplify the deployment of services and functions to Knative. 1. **`serverless.yml`** (for [functions only](#deploying-functions)): When using serverless to deploy functions, the `serverless.yml` file - will contain the information for all the functions being hosted in the repository as well as a reference to the - runtime being used. + will contain the information for all the functions being hosted in the repository as well as a reference to the + runtime being used. 1. **`Dockerfile`** (for [applications only](#deploying-serverless-applications): Knative requires a - `Dockerfile` in order to build your applications. It should be included at the root of your - project's repo and expose port `8080`. `Dockerfile` is not require if you plan to build serverless functions - using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). + `Dockerfile` in order to build your applications. It should be included at the root of your + project's repo and expose port `8080`. `Dockerfile` is not require if you plan to build serverless functions + using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). 1. **Prometheus** (optional): Installing Prometheus allows you to monitor the scale and traffic of your serverless function/application. - See [Installing Applications](../index.md#installing-applications) for more information. + See [Installing Applications](../index.md#installing-applications) for more information. ## Installing Knative via GitLab's Kubernetes integration @@ -60,9 +59,9 @@ The minimum recommended cluster size to run Knative is 3-nodes, 6 vCPUs, and 22. 1. [Add a Kubernetes cluster](../index.md) and [install Helm](../index.md#installing-applications). 1. Once Helm has been successfully installed, scroll down to the Knative app section. Enter the domain to be used with - your application/functions (e.g. `example.com`) and click **Install**. + your application/functions (e.g. `example.com`) and click **Install**. -  +  1. After the Knative installation has finished, you can wait for the IP address or hostname to be displayed in the **Knative Endpoint** field or [retrieve the Istio Ingress Endpoint manually](../#manually-determining-the-external-endpoint). @@ -77,7 +76,7 @@ The minimum recommended cluster size to run Knative is 3-nodes, 6 vCPUs, and 22. if your Knative base domain is `knative.info` then you need to create an A record or CNAME record with domain `*.knative.info` pointing the ip address or hostname of the ingress. -  +  NOTE: **Note:** You can deploy either [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) @@ -97,51 +96,58 @@ cluster which already has Knative installed. You must do the following: 1. Follow the steps to - [add an existing Kubernetes cluster](../index.md#adding-an-existing-kubernetes-cluster). + [add an existing Kubernetes + cluster](../index.md#add-existing-kubernetes-cluster). 1. Ensure GitLab can manage Knative: - - For a non-GitLab managed cluster, ensure that the service account for the token - provided can manage resources in the `serving.knative.dev` API group. - - For a GitLab managed cluster, - GitLab uses a service account with the `edit` cluster role. This account needs - the ability to manage resources in the `serving.knative.dev` API group. - We suggest you do this with an [aggregated ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) - adding rules to the default `edit` cluster role: - First, save the following YAML as `knative-serving-only-role.yaml`: - - ```yaml - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: knative-serving-only-role - labels: - rbac.authorization.k8s.io/aggregate-to-edit: "true" - rules: - - apiGroups: - - serving.knative.dev - resources: - - configurations - - configurationgenerations - - routes - - revisions - - revisionuids - - autoscalers - - services - verbs: - - get - - list - - create - - update - - delete - - patch - - watch - ``` - - Then run the following command: - - ```bash - kubectl apply -f knative-serving-only-role.yaml - ``` + - For a non-GitLab managed cluster, ensure that the service account for the token + provided can manage resources in the `serving.knative.dev` API group. + - For a GitLab managed cluster, if you added the cluster in [GitLab 12.1 or later](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30235), + then GitLab will already have the required access and you can proceed to the next step. + + Otherwise, you need to manually grant GitLab's service account the ability to manage + resources in the `serving.knative.dev` API group. Since every GitLab service account + has the `edit` cluster role, the simplest way to do this is with an + [aggregated ClusterRole](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#aggregated-clusterroles) + adding rules to the default `edit` cluster role: First, save the following YAML as + `knative-serving-only-role.yaml`: + + ```yaml + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: knative-serving-only-role + labels: + rbac.authorization.k8s.io/aggregate-to-edit: "true" + rules: + - apiGroups: + - serving.knative.dev + resources: + - configurations + - configurationgenerations + - routes + - revisions + - revisionuids + - autoscalers + - services + verbs: + - get + - list + - create + - update + - delete + - patch + - watch + ``` + + Then run the following command: + + ```bash + kubectl apply -f knative-serving-only-role.yaml + ``` + + If you would rather grant permissions on a per service account basis, you can do this + using a `Role` and `RoleBinding` specific to the service account and namespace. 1. Follow the steps to deploy [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) onto your @@ -151,9 +157,9 @@ You must do the following: > Introduced in GitLab 11.6. -Using functions is useful for dealing with independent -events without needing to maintain a complex unified infrastructure. This allows -you to focus on a single task that can be executed/scaled automatically and independently. +Using functions is useful for dealing with independent events without needing +to maintain a complex unified infrastructure. This allows you to focus on a +single task that can be executed/scaled automatically and independently. Currently the following [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) are offered: @@ -161,15 +167,21 @@ Currently the following [runtimes](https://gitlab.com/gitlab-org/serverless/runt - node.js - Dockerfile -You can find and import all the files referenced in this doc in the **[functions example project](https://gitlab.com/knative-examples/functions)**. +`Dockerfile` presence is assumed when a runtime is not specified. -Follow these steps to deploy a function using the Node.js runtime to your Knative instance (you can skip these steps if you've cloned the example project): +You can find and import all the files referenced in this doc in the +**[functions example project](https://gitlab.com/knative-examples/functions)**. -1. Create a directory that will house the function. In this example we will create a directory called `echo` at the root of the project. +Follow these steps to deploy a function using the Node.js runtime to your +Knative instance (you can skip these steps if you've cloned the example +project): + +1. Create a directory that will house the function. In this example we will + create a directory called `echo` at the root of the project. 1. Create the file that will contain the function code. In this example, our file is called `echo.js` and is located inside the `echo` directory. If your project is: - - Public, continue to the next step. - - Private, you will need to [create a GitLab deploy token](../../deploy_tokens/index.md#creating-a-deploy-token) with `gitlab-deploy-token` as the name and the `read_registry` scope. + - Public, continue to the next step. + - Private, you will need to [create a GitLab deploy token](../../deploy_tokens/index.md#creating-a-deploy-token) with `gitlab-deploy-token` as the name and the `read_registry` scope. 1. `.gitlab-ci.yml`: this defines a pipeline used to deploy your functions. It must be included at the root of your repository: @@ -187,16 +199,16 @@ Follow these steps to deploy a function using the Node.js runtime to your Knativ environment: production ``` - This `.gitlab-ci.yml` creates jobs that invoke some predefined commands to - build and deploy your functions to your cluster. + This `.gitlab-ci.yml` creates jobs that invoke some predefined commands to + build and deploy your functions to your cluster. - `Serverless.gitlab-ci.yml` is a template that allows customization. - You can either import it with `include` parameter and use `extends` to - customize your jobs, or you can inline the entire template by choosing it - from **Apply a template** dropdown when editing the `.gitlab-ci.yml` file through - the user interface. + `Serverless.gitlab-ci.yml` is a template that allows customization. + You can either import it with `include` parameter and use `extends` to + customize your jobs, or you can inline the entire template by choosing it + from **Apply a template** dropdown when editing the `.gitlab-ci.yml` file through + the user interface. -2. `serverless.yml`: this file contains the metadata for your functions, +1. `serverless.yml`: this file contains the metadata for your functions, such as name, runtime, and environment. It must be included at the root of your repository. @@ -237,26 +249,27 @@ Explanation of the fields used above: | Parameter | Description | |-----------|-------------| -| `name` | Indicates which provider is used to execute the `serverless.yml` file. In this case, the TriggerMesh `tm` CLI. | +| `name` | Indicates which provider is used to execute the `serverless.yml` file. In this case, the TriggerMesh middleware. | | `environment` | Includes the environment variables to be passed as part of function execution for **all** functions in the file, where `FOO` is the variable name and `BAR` are he variable contents. You may replace this with you own variables. | ### `functions` -In the `serverless.yml` example above, the function name is `echo` and the subsequent lines contain the function attributes. +In the `serverless.yml` example above, the function name is `echo` and the +subsequent lines contain the function attributes. | Parameter | Description | |-----------|-------------| | `handler` | The function's name. | | `source` | Directory with sources of a functions. | -| `runtime` | The runtime to be used to execute the function. | +| `runtime` (optional)| The runtime to be used to execute the function. When the runtime is not specified, we assume that `Dockerfile` is present in the function directory specified by `source`. | | `description` | A short description of the function. | | `environment` | Sets an environment variable for the specific function only. | -After the `gitlab-ci.yml` template has been added and the `serverless.yml` file has been -created, pushing a commit to your project will result in a -CI pipeline being executed which will deploy each function as a Knative service. -Once the deploy stage has finished, additional details for the function will -appear under **Operations > Serverless**. +After the `gitlab-ci.yml` template has been added and the `serverless.yml` file +has been created, pushing a commit to your project will result in a CI pipeline +being executed which will deploy each function as a Knative service. Once the +deploy stage has finished, additional details for the function will appear +under **Operations > Serverless**.  @@ -275,16 +288,17 @@ The sample function can now be triggered from any HTTP client using a simple `PO 1. Using curl (replace the URL on the last line with the URL of your application): - ```bash - curl \ - --header "Content-Type: application/json" \ - --request POST \ - --data '{"GitLab":"FaaS"}' \ - http://functions-echo.functions-1.functions.example.com/ - ``` - 2. Using a web-based tool (ie. postman, restlet, etc) + ```bash + curl \ + --header "Content-Type: application/json" \ + --request POST \ + --data '{"GitLab":"FaaS"}' \ + http://functions-echo.functions-1.functions.example.com/ + ``` + + 1. Using a web-based tool (ie. postman, restlet, etc) -  +  ## Deploying Serverless applications @@ -313,6 +327,8 @@ customize your jobs, or you can inline the entire template by choosing it from **Apply a template** dropdown when editing the `.gitlab-ci.yml` file through the user interface. +A `serverless.yml` file is not required when deploying serverless applications. + ### Deploy the application with Knative With all the pieces in place, the next time a CI pipeline runs, the Knative application will be deployed. Navigate to @@ -327,27 +343,23 @@ Go to the **CI/CD > Pipelines** and click on the pipeline that deployed your app The output will look like this: ```bash -Running with gitlab-runner 11.5.0~beta.844.g96d88322 (96d88322) - on docker-auto-scale 72989761 -Using Docker executor with image gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 ... -Pulling docker image gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 ... -Using docker image sha256:6b3f6590a9b30bd7aafb9573f047d930c70066e43955b4beb18a1eee175f6de1 for gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 ... -Running on runner-72989761-project-4342902-concurrent-0 via runner-72989761-stg-srm-1541795796-27929c96... -Cloning repository... -Cloning into '/builds/danielgruesso/knative'... -Checking out 8671ad20 as master... -Skipping Git submodules setup -$ echo "$CI_REGISTRY_IMAGE" -registry.staging.gitlab.com/danielgruesso/knative -$ tm -n "$KUBE_NAMESPACE" --config "$KUBECONFIG" deploy service "$CI_PROJECT_NAME" --from-image "$CI_REGISTRY_IMAGE" --wait -Deployment started. Run "tm -n knative-4342902 describe service knative" to see the details -Waiting for ready state....... -Service domain: knative.knative-4342902.example.com +Running with gitlab-runner 12.1.0-rc1 (6da35412) + on prm-com-gitlab-org ae3bfce3 +Using Docker executor with image registry.gitlab.com/gitlab-org/gitlabktl:latest ... +Running on runner-ae3bfc-concurrent-0 via runner-ae3bfc ... +Fetching changes... +Authenticating with credentials from job payload (GitLab Registry) +$ /usr/bin/gitlabktl application deploy +Welcome to gitlabktl tool +time="2019-07-15T10:51:07Z" level=info msg="deploying registry credentials" +Creating app-hello function +Waiting for app-hello ready state +Service app-hello URL: http://app-hello.serverless.example.com Job succeeded ``` -The second to last line, labeled **Service domain** contains the URL for the deployment. Copy and paste the domain into your -browser to see the app live. +The second to last line, labeled **Service domain** contains the URL for the +deployment. Copy and paste the domain into your browser to see the app live.  @@ -376,269 +388,268 @@ cluster. By default, a GitLab serverless deployment will be served over `http`. In order to serve over `https` you must manually obtain and install TLS certificates. -The simplest way to accomplish this is to +The simplest way to accomplish this is to use [Certbot to manually obtain Let's Encrypt certificates](https://knative.dev/docs/serving/using-a-tls-cert/#using-certbot-to-manually-obtain-let-s-encrypt-certificates). Certbot is a free, open source software tool for automatically using Let’s Encrypt certificates on manually-administrated websites to enable HTTPS. NOTE: **Note:** The instructions below relate to installing and running Certbot on a Linux server and may not work on other operating systems. -1. Install Certbot by running the +1. Install Certbot by running the [`certbot-auto` wrapper script](https://certbot.eff.org/docs/install.html#certbot-auto). On the command line of your server, run the following commands: - ```sh - wget https://dl.eff.org/certbot-auto - sudo mv certbot-auto /usr/local/bin/certbot-auto - sudo chown root /usr/local/bin/certbot-auto - chmod 0755 /usr/local/bin/certbot-auto - /usr/local/bin/certbot-auto --help - ``` - - To check the integrity of the `certbot-auto` script, run: - - ```sh - wget -N https://dl.eff.org/certbot-auto.asc - gpg2 --keyserver ipv4.pool.sks-keyservers.net --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 - gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc /usr/local/bin/certbot-auto - ``` - - The output of the last command should look something like: - - ```sh - gpg: Signature made Mon 10 Jun 2019 06:24:40 PM EDT - gpg: using RSA key A2CFB51FA275A7286234E7B24D17C995CD9775F2 - gpg: key 4D17C995CD9775F2 marked as ultimately trusted - gpg: checking the trustdb - gpg: marginals needed: 3 completes needed: 1 trust model: pgp - gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u - gpg: next trustdb check due at 2027-11-22 - gpg: Good signature from "Let's Encrypt Client Team <letsencrypt-client@eff.org>" [ultimate] - ``` + ```sh + wget https://dl.eff.org/certbot-auto + sudo mv certbot-auto /usr/local/bin/certbot-auto + sudo chown root /usr/local/bin/certbot-auto + chmod 0755 /usr/local/bin/certbot-auto + /usr/local/bin/certbot-auto --help + ``` + + To check the integrity of the `certbot-auto` script, run: + + ```sh + wget -N https://dl.eff.org/certbot-auto.asc + gpg2 --keyserver ipv4.pool.sks-keyservers.net --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc /usr/local/bin/certbot-auto + ``` + + The output of the last command should look something like: + + ```sh + gpg: Signature made Mon 10 Jun 2019 06:24:40 PM EDT + gpg: using RSA key A2CFB51FA275A7286234E7B24D17C995CD9775F2 + gpg: key 4D17C995CD9775F2 marked as ultimately trusted + gpg: checking the trustdb + gpg: marginals needed: 3 completes needed: 1 trust model: pgp + gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u + gpg: next trustdb check due at 2027-11-22 + gpg: Good signature from "Let's Encrypt Client Team <letsencrypt-client@eff.org>" [ultimate] + ``` 1. Run the following command to use Certbot to request a certificate using DNS challenge during authorization: + ```sh + ./certbot-auto certonly --manual --preferred-challenges dns -d '*.<namespace>.example.com' + ``` + + Where `<namespace>` is the namespace created by GitLab for your serverless project (composed of `<project_name>-<project_id>-<environment>`) and + `example.com` is the domain being used for your project. If you are unsure what the namespace of your project is, navigate + to the **Operations > Serverless** page of your project and inspect + the endpoint provided for your function/app. + +  + + In the above image, the namespace for the project is `node-function-11909507` and the domain is `knative.info`, thus + certificate request line would look like this: + + ```sh + ./certbot-auto certonly --manual --preferred-challenges dns -d '*.node-function-11909507.knative.info' + ``` - ```sh - ./certbot-auto certonly --manual --preferred-challenges dns -d '*.<namespace>.example.com' - ``` - - Where `<namespace>` is the namespace created by GitLab for your serverless project (composed of `<projectname+id>`) and - `example.com` is the domain being used for your project. If you are unsure what the namespace of your project is, navigate - to the **Operations > Serverless** page of your project and inspect - the endpoint provided for your function/app. - -  - - In the above image, the namespace for the project is `node-function-11909507` and the domain is `knative.info`, thus - certificate request line would look like this: - - ```sh - ./certbot-auto certonly --manual --preferred-challenges dns -d '*.node-function-11909507.knative.info' - ``` - - The Certbot tool walks you through the steps of validating that you own each domain that you specify by creating TXT records in those domains. - After this process is complete, the output should look something like this: - - ```sh - IMPORTANT NOTES: - - Congratulations! Your certificate and chain have been saved at: - /etc/letsencrypt/live/namespace.example.com/fullchain.pem - Your key file has been saved at: - /etc/letsencrypt/live/namespace.example/privkey.pem - Your cert will expire on 2019-09-19. To obtain a new or tweaked - version of this certificate in the future, simply run certbot-auto - again. To non-interactively renew *all* of your certificates, run - "certbot-auto renew" - -----BEGIN PRIVATE KEY----- - - Your account credentials have been saved in your Certbot - configuration directory at /etc/letsencrypt. You should make a - secure backup of this folder now. This configuration directory will - also contain certificates and private keys obtained by Certbot so - making regular backups of this folder is ideal. - ``` + The Certbot tool walks you through the steps of validating that you own each domain that you specify by creating TXT records in those domains. + After this process is complete, the output should look something like this: + + ```sh + IMPORTANT NOTES: + - Congratulations! Your certificate and chain have been saved at: + /etc/letsencrypt/live/namespace.example.com/fullchain.pem + Your key file has been saved at: + /etc/letsencrypt/live/namespace.example/privkey.pem + Your cert will expire on 2019-09-19. To obtain a new or tweaked + version of this certificate in the future, simply run certbot-auto + again. To non-interactively renew *all* of your certificates, run + "certbot-auto renew" + -----BEGIN PRIVATE KEY----- + - Your account credentials have been saved in your Certbot + configuration directory at /etc/letsencrypt. You should make a + secure backup of this folder now. This configuration directory will + also contain certificates and private keys obtained by Certbot so + making regular backups of this folder is ideal. + ``` 1. Create certificate and private key files. Using the contents of the files returned by Certbot, we'll create two files in order to create the Kubernetes secret: - Run the following command to see the contents of `fullchain.pem`: - - ```sh - sudo cat /etc/letsencrypt/live/node-function-11909507.knative.info/fullchain.pem - ``` - - Output should look like this: - - ```sh - -----BEGIN CERTIFICATE----- - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b4ag== - -----END CERTIFICATE----- - -----BEGIN CERTIFICATE----- - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - K2fcb195768c39e9a94cec2c2e30Qg== - -----END CERTIFICATE----- - ``` - - Create a file with the name `cert.pem` with the contents of the entire output. - - Once `cert.pem` is created, run the following command to see the contents of `privkey.pem`: - - ```sh - sudo cat /etc/letsencrypt/live/namespace.example/privkey.pem - ``` - - Output should look like this: - - ```sh - -----BEGIN PRIVATE KEY----- - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df - -----BEGIN CERTIFICATE----- - fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 - 4f294d1eaca42b8692017b4262== - -----END PRIVATE KEY----- - ``` - - Create a new file with the name `cert.pk` with the contents of the entire output. + Run the following command to see the contents of `fullchain.pem`: + + ```sh + sudo cat /etc/letsencrypt/live/node-function-11909507.knative.info/fullchain.pem + ``` + + Output should look like this: + + ```sh + -----BEGIN CERTIFICATE----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b4ag== + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + K2fcb195768c39e9a94cec2c2e30Qg== + -----END CERTIFICATE----- + ``` + + Create a file with the name `cert.pem` with the contents of the entire output. + + Once `cert.pem` is created, run the following command to see the contents of `privkey.pem`: + + ```sh + sudo cat /etc/letsencrypt/live/namespace.example/privkey.pem + ``` + + Output should look like this: + + ```sh + -----BEGIN PRIVATE KEY----- + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + 2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df + -----BEGIN CERTIFICATE----- + fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6 + 4f294d1eaca42b8692017b4262== + -----END PRIVATE KEY----- + ``` + + Create a new file with the name `cert.pk` with the contents of the entire output. 1. Create a Kubernetes secret to hold your TLS certificate, `cert.pem`, and the private key `cert.pk`: - NOTE: **Note:** - Running `kubectl` commands on your cluster requires setting up access to the cluster first. - For clusters created on GKE, see - [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl). - For other platforms, [install `kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/). - - ```sh - kubectl create --namespace istio-system secret tls istio-ingressgateway-certs \ - --key cert.pk \ - --cert cert.pem - ``` + NOTE: **Note:** + Running `kubectl` commands on your cluster requires setting up access to the cluster first. + For clusters created on GKE, see + [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl). + For other platforms, [install `kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + + ```sh + kubectl create --namespace istio-system secret tls istio-ingressgateway-certs \ + --key cert.pk \ + --cert cert.pem + ``` - Where `cert.pem` and `cert.pk` are your certificate and private key files. Note that the `istio-ingressgateway-certs` secret name is required. + Where `cert.pem` and `cert.pk` are your certificate and private key files. Note that the `istio-ingressgateway-certs` secret name is required. 1. Configure Knative to use the new secret that you created for HTTPS - connections. Run the - following command to open the Knative shared `gateway` in edit mode: - - ```sh - kubectl edit gateway knative-ingress-gateway --namespace knative-serving - ``` - - Update the gateway to include the following tls: section and configuration: - - ```sh - tls: - mode: SIMPLE - privateKey: /etc/istio/ingressgateway-certs/tls.key - serverCertificate: /etc/istio/ingressgateway-certs/tls.crt - ``` - - Example: - - ```sh - apiVersion: networking.istio.io/v1alpha3 - kind: Gateway - metadata: - # ... skipped ... - spec: - selector: - istio: ingressgateway - servers: - - hosts: - - "*" - port: - name: http - number: 80 - protocol: HTTP - - hosts: - - "*" - port: - name: https - number: 443 - protocol: HTTPS - tls: - mode: SIMPLE - privateKey: /etc/istio/ingressgateway-certs/tls.key - serverCertificate: /etc/istio/ingressgateway-certs/tls.crt - ``` - - After your changes are running on your Knative cluster, you can begin using the HTTPS protocol for secure access your deployed Knative services. - In the event a mistake is made during this process and you need to update the cert, you will need to edit the gateway `knative-ingress-gateway` - to switch back to `PASSTHROUGH` mode. Once corrections are made, edit the file again so the gateway will use the new certificates.
\ No newline at end of file + connections. Run the + following command to open the Knative shared `gateway` in edit mode: + + ```sh + kubectl edit gateway knative-ingress-gateway --namespace knative-serving + ``` + + Update the gateway to include the following tls: section and configuration: + + ```sh + tls: + mode: SIMPLE + privateKey: /etc/istio/ingressgateway-certs/tls.key + serverCertificate: /etc/istio/ingressgateway-certs/tls.crt + ``` + + Example: + + ```sh + apiVersion: networking.istio.io/v1alpha3 + kind: Gateway + metadata: + # ... skipped ... + spec: + selector: + istio: ingressgateway + servers: + - hosts: + - "*" + port: + name: http + number: 80 + protocol: HTTP + - hosts: + - "*" + port: + name: https + number: 443 + protocol: HTTPS + tls: + mode: SIMPLE + privateKey: /etc/istio/ingressgateway-certs/tls.key + serverCertificate: /etc/istio/ingressgateway-certs/tls.crt + ``` + + After your changes are running on your Knative cluster, you can begin using the HTTPS protocol for secure access your deployed Knative services. + In the event a mistake is made during this process and you need to update the cert, you will need to edit the gateway `knative-ingress-gateway` + to switch back to `PASSTHROUGH` mode. Once corrections are made, edit the file again so the gateway will use the new certificates. diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index 78ffa11d59b..96c4f16fe04 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -57,8 +57,8 @@ Example `CODEOWNERS` file: # Files with a `#` can still be accesssed by escaping the pound sign \#file_with_pound.rb @owner-file-with-pound -# Multiple codeowners can be specified, separated by whitespace -CODEOWNERS @multiple @owners @tab-separated +# Multiple codeowners can be specified, separated by spaces or tabs +CODEOWNERS @multiple @code @owners # Both usernames or email addresses can be used to match # users. Everything else will be ignored. For example this will diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md index eac7fc6b195..c9eb81b990c 100644 --- a/doc/user/project/container_registry.md +++ b/doc/user/project/container_registry.md @@ -1,13 +1,8 @@ # GitLab Container Registry -> **Notes:** -> > - [Introduced][ce-4040] in GitLab 8.8. > - Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker > versions earlier than 1.10. -> - This document is about the user guide. To learn how to enable GitLab Container -> Registry across your GitLab instance, visit the -> [administrator documentation](../../administration/container_registry.md). > - Starting from GitLab 8.12, if you have 2FA enabled in your account, you need > to pass a [personal access token][pat] instead of your password in order to > login to GitLab's Container Registry. @@ -16,28 +11,33 @@ With the Docker Container Registry integrated into GitLab, every project can have its own space to store its Docker images. +This document is the user guide. To learn how to enable GitLab Container +Registry across your GitLab instance, visit the +[administrator documentation](../../administration/container_registry.md). + You can read more about Docker Registry at <https://docs.docker.com/registry/introduction/>. ## Enable the Container Registry for your project -NOTE: **Note:** -If you cannot find the Container Registry entry under your project's settings, -that means that it is not enabled in your GitLab instance. Ask your administrator -to enable it. - -1. First, ask your system administrator to enable GitLab Container Registry - following the [administration documentation](../../administration/container_registry.md). - If you are using GitLab.com, this is enabled by default so you can start using - the Registry immediately. Currently there is a soft (10GB) size restriction for - registry on GitLab.com, as part of the [repository size limit](repository/index.md). -1. Go to your [project's General settings](settings/index.md#sharing-and-permissions) +If you cannot find the **Packages > Container Registry** entry under your +project's sidebar, it is not enabled in your GitLab instance. Ask your +administrator to enable GitLab Container Registry following the +[administration documentation](../../administration/container_registry.md). + +If you are using GitLab.com, this is enabled by default so you can start using +the Registry immediately. Currently there is a soft (10GB) size restriction for +registry on GitLab.com, as part of the [repository size limit](repository/index.md). + +Once enabled for your GitLab instance, to enable Container Registry for your +project: + +1. Go to your project's **Settings > General** page. +1. Expand the **Visibility, project features, permissions** section and enable the **Container Registry** feature on your project. For new projects this might be enabled by default. For existing projects (prior GitLab 8.8), you will have to explicitly enable it. -1. Hit **Save changes** for the changes to take effect. You should now be able - to see the **Registry** link in the sidebar. - - +1. Press **Save changes** for the changes to take effect. You should now be able + to see the **Packages > Container Registry** link in the sidebar. ## Build and push images @@ -49,14 +49,14 @@ to enable it. > - To move or rename a repository with a container registry you will have to > delete all existing images. -If you visit the **Registry** link under your project's menu, you can see the -explicit instructions to login to the Container Registry using your GitLab -credentials. +If you visit the **Packages > Container Registry** link under your project's +menu, you can see the explicit instructions to login to the Container Registry +using your GitLab credentials. For example if the Registry's URL is `registry.example.com`, then you should be able to login with: -``` +```sh docker login registry.example.com ``` @@ -64,14 +64,14 @@ Building and publishing images should be a straightforward process. Just make sure that you are using the Registry URL with the namespace and project name that is hosted on GitLab: -``` +```sh docker build -t registry.example.com/group/project/image . docker push registry.example.com/group/project/image ``` Your image will be named after the following scheme: -``` +```text <registry URL>/<namespace>/<project>/<image> ``` @@ -79,7 +79,7 @@ GitLab supports up to three levels of image repository names. Following examples of image tags are valid: -``` +```text registry.example.com/group/project:some-tag registry.example.com/group/project/image:latest registry.example.com/group/project/my/image:rc1 @@ -90,7 +90,7 @@ registry.example.com/group/project/my/image:rc1 To download and run a container from images hosted in GitLab Container Registry, use `docker run`: -``` +```sh docker run [options] registry.example.com/group/project/image [arguments] ``` @@ -100,7 +100,7 @@ For more information on running Docker containers, visit the ## Control Container Registry from within GitLab GitLab offers a simple Container Registry management panel. Go to your project -and click **Registry** in the project menu. +and click **Packages > Container Registry** in the project menu. This view will show you all tags in your project and will easily allow you to delete them. @@ -173,9 +173,9 @@ curl localhost:5001/debug/vars A Docker connection error can occur when there are special characters in either the group, project or branch name. Special characters can include: -* Leading underscore -* Trailing hyphen/dash -* Double hyphen/dash +- Leading underscore +- Trailing hyphen/dash +- Double hyphen/dash To get around this, you can [change the group path](../group/index.md#changing-a-groups-path), [change the project path](../project/settings/index.md#renaming-a-repository) or chanage the branch @@ -183,7 +183,8 @@ name. ### Advanced Troubleshooting ->**NOTE:** The following section is only recommended for experts. +NOTE: **Note:** +The following section is only recommended for experts. Sometimes it's not obvious what is wrong, and you may need to dive deeper into the communication between the Docker client and the Registry to find out @@ -195,7 +196,7 @@ diagnose a problem with the S3 setup. A user attempted to enable an S3-backed Registry. The `docker login` step went fine. However, when pushing an image, the output showed: -``` +```text The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test/docker-image] dc5e59c14160: Pushing [==================================================>] 14.85 kB 03c20c1a019a: Pushing [==================================================>] 2.048 kB @@ -218,7 +219,7 @@ at the communication between the client and the Registry. The REST API between the Docker client and Registry is [described here](https://docs.docker.com/registry/spec/api/). Normally, one would just use Wireshark or tcpdump to capture the traffic and see where things went -wrong. However, since all communications between Docker clients and servers +wrong. However, since all communications between Docker clients and servers are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even if you know the private key. What can we do instead? @@ -241,10 +242,10 @@ The following installation instructions assume you are running Ubuntu: Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit. 1. Install the certificate from `~/.mitmproxy` to your system: - ```sh - sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt - sudo update-ca-certificates - ``` + ```sh + sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt + sudo update-ca-certificates + ``` If successful, the output should indicate that a certificate was added: diff --git a/doc/user/project/cycle_analytics.md b/doc/user/project/cycle_analytics.md index 5d36e1d4be3..87577c9ec88 100644 --- a/doc/user/project/cycle_analytics.md +++ b/doc/user/project/cycle_analytics.md @@ -1,161 +1,5 @@ -# Cycle Analytics - -Cycle Analytics measures the time spent to go from an [idea to production] - also known -as cycle time - for each of your projects. Cycle Analytics displays the median time for an idea to -reach production, along with the time typically spent in each DevOps stage along the way. - -Cycle Analytics is useful in order to quickly determine the velocity of a given -project. It points to bottlenecks in the development process, enabling management -to uncover, triage, and root-cause slowdowns in the software development life cycle. - -Cycle Analytics is tightly coupled with the [GitLab flow] and -calculates a separate median for each stage. - -## Overview - -You can find the Cycle Analytics page under your project's **Project ➔ Cycle -Analytics** tab. - - - -There are seven stages that are tracked as part of the Cycle Analytics calculations. - -- **Issue** (Tracker) - - Time to schedule an issue (by milestone or by adding it to an issue board) -- **Plan** (Board) - - Time to first commit -- **Code** (IDE) - - Time to create a merge request -- **Test** (CI) - - Time it takes GitLab CI/CD to test your code -- **Review** (Merge Request/MR) - - Time spent on code review -- **Staging** (Continuous Deployment) - - Time between merging and deploying to production -- **Production** (Total) - - Total lifecycle time; i.e. the velocity of the project or team - -## How the data is measured - -Cycle Analytics records cycle time and data based on the project issues with the -exception of the staging and production stages, where only data deployed to -production are measured. - -Specifically, if your CI is not set up and you have not defined a `production` -or `production/*` [environment], then you will not have any data for those stages. - -Below you can see in more detail what the various stages of Cycle Analytics mean. - -| **Stage** | **Description** | -| --------- | --------------- | -| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list][board] created for it. | -| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. | -| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern] to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the issue closing pattern is not present in the merge request description, the MR is not considered to the measurement time of the stage. | -| Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. | -| Review | Measures the median time taken to review the merge request that has closing issue pattern, between its creation and until it's merged. | -| Staging | Measures the median time between merging the merge request with closing issue pattern until the very first deployment to production. It's tracked by the [environment] set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI configuration. If there isn't a production environment, this is not tracked. | -| Production| The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. | - +--- +redirect_to: '../analytics/cycle_analytics.md' --- -Here's a little explanation of how this works behind the scenes: - -1. Issues and merge requests are grouped together in pairs, such that for each - `<issue, merge request>` pair, the merge request has the [issue closing pattern] - for the corresponding issue. All other issues and merge requests are **not** - considered. -1. Then the `<issue, merge request>` pairs are filtered out by last XX days (specified - by the UI - default is 90 days). So it prohibits these pairs from being considered. -1. For the remaining `<issue, merge request>` pairs, we check the information that - we need for the stages, like issue creation date, merge request merge time, - etc. - -To sum up, anything that doesn't follow the [GitLab flow] won't be tracked at all. -So, the Cycle Analytics dashboard won't present any data: - -- For merge requests that do not close an issue. -- For issues not labeled with a label present in the Issue Board or for issues not assigned a milestone. -- For staging and production stages, if the project has no `production` or `production/*` - environment. - -## Example workflow - -Below is a simple fictional workflow of a single cycle that happens in a -single day passing through all seven stages. Note that if a stage does not have -a start and a stop mark, it is not measured and hence not calculated in the median -time. It is assumed that milestones are created and CI for testing and setting -environments is configured. - -1. Issue is created at 09:00 (start of **Issue** stage). -1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of - **Plan** stage). -1. Start working on the issue, create a branch locally and make one commit at - 12:00. -1. Make a second commit to the branch which mentions the issue number at 12.30 - (stop of **Plan** stage / start of **Code** stage). -1. Push branch and create a merge request that contains the [issue closing pattern] - in its description at 14:00 (stop of **Code** stage / start of **Test** and - **Review** stages). -1. The CI starts running your scripts defined in [`.gitlab-ci.yml`][yml] and - takes 5min (stop of **Test** stage). -1. Review merge request, ensure that everything is OK and merge the merge - request at 19:00. (stop of **Review** stage / start of **Staging** stage). -1. Now that the merge request is merged, a deployment to the `production` - environment starts and finishes at 19:30 (stop of **Staging** stage). -1. The cycle completes and the sum of the median times of the previous stages - is recorded to the **Production** stage. That is the time between creating an - issue and deploying its relevant merge request to production. - -From the above example you can conclude the time it took each stage to complete -as long as their total time: - -- **Issue**: 2h (11:00 - 09:00) -- **Plan**: 1h (12:00 - 11:00) -- **Code**: 2h (14:00 - 12:00) -- **Test**: 5min -- **Review**: 5h (19:00 - 14:00) -- **Staging**: 30min (19:30 - 19:00) -- **Production**: Since this stage measures the sum of median time off all - previous stages, we cannot calculate it if we don't know the status of the - stages before. In case this is the very first cycle that is run in the project, - then the **Production** time is 10h 30min (19:30 - 09:00) - -A few notes: - -- In the above example we demonstrated that it doesn't matter if your first - commit doesn't mention the issue number, you can do this later in any commit - of the branch you are working on. -- You can see that the **Test** stage is not calculated to the overall time of - the cycle since it is included in the **Review** process (every MR should be - tested). -- The example above was just **one cycle** of the seven stages. Add multiple - cycles, calculate their median time and the result is what the dashboard of - Cycle Analytics is showing. - -## Permissions - -The current permissions on the Cycle Analytics dashboard are: - -- Public projects - anyone can access -- Internal projects - any authenticated user can access -- Private projects - any member Guest and above can access - -You can [read more about permissions][permissions] in general. - -## More resources - -Learn more about Cycle Analytics in the following resources: - -- [Cycle Analytics feature page](https://about.gitlab.com/features/cycle-analytics/) -- [Cycle Analytics feature preview](https://about.gitlab.com/2016/09/16/feature-preview-introducing-cycle-analytics/) -- [Cycle Analytics feature highlight](https://about.gitlab.com/2016/09/21/cycle-analytics-feature-highlight/) - -[board]: issue_board.md#creating-a-new-list -[ce-5986]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5986 -[ce-20975]: https://gitlab.com/gitlab-org/gitlab-ce/issues/20975 -[environment]: ../../ci/yaml/README.md#environment -[GitLab flow]: ../../workflow/gitlab_flow.md -[idea to production]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab -[issue closing pattern]: issues/managing_issues.md#closing-issues-automatically -[permissions]: ../permissions.md -[yml]: ../../ci/yaml/README.md +This document was moved to [another location](../analytics/cycle_analytics.md) diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index cb1faa771bc..0a83f2e894a 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -63,12 +63,12 @@ To display the Deploy Boards for a specific [environment] you should: 1. Have a Kubernetes cluster up and running. - NOTE: **Running on OpenShift:** - If you are using OpenShift, ensure that you're using the `Deployment` resource - instead of `DeploymentConfiguration`, otherwise the Deploy Boards won't render - correctly. For more information, read the - [OpenShift docs](https://docs.openshift.com/container-platform/3.7/dev_guide/deployments/kubernetes_deployments.html#kubernetes-deployments-vs-deployment-configurations) - and [GitLab issue #4584](https://gitlab.com/gitlab-org/gitlab-ee/issues/4584). + NOTE: **Running on OpenShift:** + If you are using OpenShift, ensure that you're using the `Deployment` resource + instead of `DeploymentConfiguration`, otherwise the Deploy Boards won't render + correctly. For more information, read the + [OpenShift docs](https://docs.openshift.com/container-platform/3.7/dev_guide/deployments/kubernetes_deployments.html#kubernetes-deployments-vs-deployment-configurations) + and [GitLab issue #4584](https://gitlab.com/gitlab-org/gitlab-ee/issues/4584). 1. [Configure GitLab Runner][runners] with the [Docker][docker-exec] or [Kubernetes][kube-exec] executor. @@ -93,7 +93,7 @@ To display the Deploy Boards for a specific [environment] you should: To migrate, please apply the required annotations (see above) and re-deploy your application. -  +  Once all of the above are set up and the pipeline has run at least once, navigate to the environments page under **Operations > Environments**. diff --git a/doc/user/project/deploy_tokens/img/deploy_tokens.png b/doc/user/project/deploy_tokens/img/deploy_tokens.png Binary files differindex 421aa1ab3e5..493de8e0fce 100644 --- a/doc/user/project/deploy_tokens/img/deploy_tokens.png +++ b/doc/user/project/deploy_tokens/img/deploy_tokens.png diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 72594733cd3..164d1dddff0 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -56,9 +56,9 @@ To download a repository using a Deploy Token, you just need to: 1. Take note of your `username` and `token`. 1. `git clone` the project using the Deploy Token: - ```sh - git clone http://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git - ``` + ```sh + git clone http://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git + ``` Replace `<username>` and `<deploy_token>` with the proper values. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 196874fdc86..f53dc056010 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -55,7 +55,7 @@ changes you made after picking the template and return it to its initial status.  -## Setting a default template for issues and merge requests **(STARTER)** +## Setting a default template for merge requests and issues **(STARTER)** > **Notes:** > @@ -66,20 +66,20 @@ changes you made after picking the template and return it to its initial status. > - Templates for merge requests were [introduced][ee-7478ece] in GitLab EE 6.9. The visibility of issues and/or merge requests should be set to either "Everyone -with access" or "Only Project Members" in your project's **Settings** otherwise the +with access" or "Only Project Members" in your project's **Settings / Visibility, project features, permissions** section, otherwise the template text areas won't show. This is the default behavior so in most cases you should be fine. -Go to your project's **Settings** and fill in the "Default description template -for issues" and "Default description template for merge requests" text areas -for issues and merge requests respectively. Since GitLab issues and merge -request support [Markdown](../markdown.md), you can use special markup like +Go to your project's **Settings** and under the **Merge requests** header, click *Expand* and fill in the "Default description template +for merge requests" text area. Under the **Default issue template**, click *Expand* and fill in "Default description template for issues" text area. Since GitLab merge request and issues + support [Markdown](../markdown.md), you can use special markup like headings, lists, etc. - + + After you add the description, hit **Save changes** for the settings to take -effect. Now, every time a new issue or merge request is created, it will be +effect. Now, every time a new merge request or issue is created, it will be pre-filled with the text you entered in the template(s). ## Description template example diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index dec679fc975..dce8c3a97bc 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -40,7 +40,7 @@ To lock a file: 1. Pick the file you want to lock. 1. Click the "Lock" button. -  +  To lock an entire directory, look for the "Lock" link next to "History". diff --git a/doc/user/project/img/autocomplete_characters_example1_v12_0.png b/doc/user/project/img/autocomplete_characters_example1_v12_0.png Binary files differnew file mode 100644 index 00000000000..9c6fa923b80 --- /dev/null +++ b/doc/user/project/img/autocomplete_characters_example1_v12_0.png diff --git a/doc/user/project/img/autocomplete_characters_example2_v12_0.png b/doc/user/project/img/autocomplete_characters_example2_v12_0.png Binary files differnew file mode 100644 index 00000000000..b2e8a782a0b --- /dev/null +++ b/doc/user/project/img/autocomplete_characters_example2_v12_0.png diff --git a/doc/user/project/img/container_registry.png b/doc/user/project/img/container_registry.png Binary files differdeleted file mode 100644 index abbaf838538..00000000000 --- a/doc/user/project/img/container_registry.png +++ /dev/null diff --git a/doc/user/project/img/cycle_analytics_landing_page.png b/doc/user/project/img/cycle_analytics_landing_page.png Binary files differdeleted file mode 100644 index cf46098b9a4..00000000000 --- a/doc/user/project/img/cycle_analytics_landing_page.png +++ /dev/null diff --git a/doc/user/project/img/deploy_boards_kubernetes_label.png b/doc/user/project/img/deploy_boards_kubernetes_label.png Binary files differindex 19785c74d07..130ff2caa8a 100644 --- a/doc/user/project/img/deploy_boards_kubernetes_label.png +++ b/doc/user/project/img/deploy_boards_kubernetes_label.png diff --git a/doc/user/project/img/description_templates_default_settings.png b/doc/user/project/img/description_templates_default_settings.png Binary files differdeleted file mode 100644 index ab314e83d06..00000000000 --- a/doc/user/project/img/description_templates_default_settings.png +++ /dev/null diff --git a/doc/user/project/img/description_templates_issue_settings.png b/doc/user/project/img/description_templates_issue_settings.png Binary files differnew file mode 100644 index 00000000000..53328108835 --- /dev/null +++ b/doc/user/project/img/description_templates_issue_settings.png diff --git a/doc/user/project/img/description_templates_merge_request_settings.png b/doc/user/project/img/description_templates_merge_request_settings.png Binary files differnew file mode 100644 index 00000000000..eda264f7f37 --- /dev/null +++ b/doc/user/project/img/description_templates_merge_request_settings.png diff --git a/doc/user/project/img/file_lock.png b/doc/user/project/img/file_lock.png Binary files differindex 82699a12ffd..e881442630b 100644 --- a/doc/user/project/img/file_lock.png +++ b/doc/user/project/img/file_lock.png diff --git a/doc/user/project/img/file_lock_merge_request_error_message.png b/doc/user/project/img/file_lock_merge_request_error_message.png Binary files differindex 4ef04b15bef..64bcc86ac0d 100644 --- a/doc/user/project/img/file_lock_merge_request_error_message.png +++ b/doc/user/project/img/file_lock_merge_request_error_message.png diff --git a/doc/user/project/img/file_lock_repository_view.png b/doc/user/project/img/file_lock_repository_view.png Binary files differindex a2cab0decab..ced14198da9 100644 --- a/doc/user/project/img/file_lock_repository_view.png +++ b/doc/user/project/img/file_lock_repository_view.png diff --git a/doc/user/project/img/issue_boards_multiple.png b/doc/user/project/img/issue_boards_multiple.png Binary files differindex 4b1a8356dc9..e6183360610 100644 --- a/doc/user/project/img/issue_boards_multiple.png +++ b/doc/user/project/img/issue_boards_multiple.png diff --git a/doc/user/project/img/key_value_labels.png b/doc/user/project/img/key_value_labels.png Binary files differdeleted file mode 100644 index bec901f127f..00000000000 --- a/doc/user/project/img/key_value_labels.png +++ /dev/null diff --git a/doc/user/project/img/labels_default.png b/doc/user/project/img/labels_default.png Binary files differdeleted file mode 100644 index 221c5cf55a2..00000000000 --- a/doc/user/project/img/labels_default.png +++ /dev/null diff --git a/doc/user/project/img/labels_default_v12_1.png b/doc/user/project/img/labels_default_v12_1.png Binary files differnew file mode 100644 index 00000000000..b36b5dac80b --- /dev/null +++ b/doc/user/project/img/labels_default_v12_1.png diff --git a/doc/user/project/img/labels_delete_v12_1.png b/doc/user/project/img/labels_delete_v12_1.png Binary files differnew file mode 100644 index 00000000000..566e0519fbe --- /dev/null +++ b/doc/user/project/img/labels_delete_v12_1.png diff --git a/doc/user/project/img/labels_drag_priority_v12_1.gif b/doc/user/project/img/labels_drag_priority_v12_1.gif Binary files differnew file mode 100644 index 00000000000..a568490da5f --- /dev/null +++ b/doc/user/project/img/labels_drag_priority_v12_1.gif diff --git a/doc/user/project/img/labels_epic_sidebar.png b/doc/user/project/img/labels_epic_sidebar.png Binary files differdeleted file mode 100644 index f8162d89e9d..00000000000 --- a/doc/user/project/img/labels_epic_sidebar.png +++ /dev/null diff --git a/doc/user/project/img/labels_epic_sidebar_v12_1.png b/doc/user/project/img/labels_epic_sidebar_v12_1.png Binary files differnew file mode 100644 index 00000000000..13fbab33e26 --- /dev/null +++ b/doc/user/project/img/labels_epic_sidebar_v12_1.png diff --git a/doc/user/project/img/labels_generate_default.png b/doc/user/project/img/labels_generate_default.png Binary files differdeleted file mode 100644 index 982a4df999c..00000000000 --- a/doc/user/project/img/labels_generate_default.png +++ /dev/null diff --git a/doc/user/project/img/labels_generate_default_v12_1.png b/doc/user/project/img/labels_generate_default_v12_1.png Binary files differnew file mode 100644 index 00000000000..cbdda2ab4dd --- /dev/null +++ b/doc/user/project/img/labels_generate_default_v12_1.png diff --git a/doc/user/project/img/labels_group_issues.png b/doc/user/project/img/labels_group_issues.png Binary files differdeleted file mode 100644 index cea1d304d31..00000000000 --- a/doc/user/project/img/labels_group_issues.png +++ /dev/null diff --git a/doc/user/project/img/labels_group_issues_v12_1.png b/doc/user/project/img/labels_group_issues_v12_1.png Binary files differnew file mode 100644 index 00000000000..3f8f93c1dfd --- /dev/null +++ b/doc/user/project/img/labels_group_issues_v12_1.png diff --git a/doc/user/project/img/labels_key_value_v12_1.png b/doc/user/project/img/labels_key_value_v12_1.png Binary files differnew file mode 100644 index 00000000000..52906b7b458 --- /dev/null +++ b/doc/user/project/img/labels_key_value_v12_1.png diff --git a/doc/user/project/img/labels_list.png b/doc/user/project/img/labels_list.png Binary files differdeleted file mode 100644 index 6940dde68bf..00000000000 --- a/doc/user/project/img/labels_list.png +++ /dev/null diff --git a/doc/user/project/img/labels_list_v12_1.png b/doc/user/project/img/labels_list_v12_1.png Binary files differnew file mode 100644 index 00000000000..47359d05f7f --- /dev/null +++ b/doc/user/project/img/labels_list_v12_1.png diff --git a/doc/user/project/img/new_label_from_sidebar.gif b/doc/user/project/img/labels_new_label_from_sidebar.gif Binary files differindex 572b29a86e1..572b29a86e1 100644 --- a/doc/user/project/img/new_label_from_sidebar.gif +++ b/doc/user/project/img/labels_new_label_from_sidebar.gif diff --git a/doc/user/project/img/labels_prioritized.png b/doc/user/project/img/labels_prioritized.png Binary files differdeleted file mode 100644 index 7ce2d08b38c..00000000000 --- a/doc/user/project/img/labels_prioritized.png +++ /dev/null diff --git a/doc/user/project/img/labels_prioritized_v12_1.png b/doc/user/project/img/labels_prioritized_v12_1.png Binary files differnew file mode 100644 index 00000000000..512c5d59a5a --- /dev/null +++ b/doc/user/project/img/labels_prioritized_v12_1.png diff --git a/doc/user/project/img/labels_promotion.png b/doc/user/project/img/labels_promotion.png Binary files differdeleted file mode 100644 index 762a3773692..00000000000 --- a/doc/user/project/img/labels_promotion.png +++ /dev/null diff --git a/doc/user/project/img/labels_promotion_v12_1.png b/doc/user/project/img/labels_promotion_v12_1.png Binary files differnew file mode 100644 index 00000000000..7cd8ff24e02 --- /dev/null +++ b/doc/user/project/img/labels_promotion_v12_1.png diff --git a/doc/user/project/img/labels_subscriptions.png b/doc/user/project/img/labels_subscriptions.png Binary files differdeleted file mode 100644 index f3c4235d051..00000000000 --- a/doc/user/project/img/labels_subscriptions.png +++ /dev/null diff --git a/doc/user/project/img/labels_subscriptions_v12_1.png b/doc/user/project/img/labels_subscriptions_v12_1.png Binary files differnew file mode 100644 index 00000000000..fa83b7db414 --- /dev/null +++ b/doc/user/project/img/labels_subscriptions_v12_1.png diff --git a/doc/user/project/img/service_desk_disabled.png b/doc/user/project/img/service_desk_disabled.png Binary files differindex 3ae64dcbe96..edae7e76986 100644 --- a/doc/user/project/img/service_desk_disabled.png +++ b/doc/user/project/img/service_desk_disabled.png diff --git a/doc/user/project/img/service_desk_enabled.png b/doc/user/project/img/service_desk_enabled.png Binary files differindex 329348e4b52..9c143ff58cd 100644 --- a/doc/user/project/img/service_desk_enabled.png +++ b/doc/user/project/img/service_desk_enabled.png diff --git a/doc/user/project/img/service_desk_issue_tracker.png b/doc/user/project/img/service_desk_issue_tracker.png Binary files differindex 485751fab0a..02d18c9debb 100644 --- a/doc/user/project/img/service_desk_issue_tracker.png +++ b/doc/user/project/img/service_desk_issue_tracker.png diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 1cc3bd3363d..e509e333313 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -31,12 +31,12 @@ to enable this if not already. ## How it works When issues/pull requests are being imported, the Bitbucket importer tries to find -the Bitbucket author/assignee in GitLab's database using the Bitbucket ID. For this -to work, the Bitbucket author/assignee should have signed in beforehand in GitLab -and **associated their Bitbucket account**. If the user is not -found in GitLab's database, the project creator (most of the times the current -user that started the import process) is set as the author, but a reference on -the issue about the original Bitbucket author is kept. +the Bitbucket author/assignee in GitLab's database using the Bitbucket `nickname`. +For this to work, the Bitbucket author/assignee should have signed in beforehand in GitLab +and **associated their Bitbucket account**. Their `nickname` must also match their Bitbucket +`username.`. If the user is not found in GitLab's database, the project creator +(most of the times the current user that started the import process) is set as the author, +but a reference on the issue about the original Bitbucket author is kept. The importer will create any new namespaces (groups) if they don't exist or in the case the namespace is taken, the repository will be imported under the user's @@ -49,17 +49,17 @@ namespace that started the import process. 1. Click on the "Bitbucket Cloud" button. -  +  1. Grant GitLab access to your Bitbucket account -  +  1. Click on the projects that you'd like to import or **Import all projects**. You can also select the namespace under which each project will be imported. -  +  [bb-import]: ../../../integration/bitbucket.md [social sign-in]: ../../profile/account/social_sign_in.md diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index d51a0c0ccca..28e211ee2ba 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -24,15 +24,16 @@ Import your projects from Bitbucket Server to GitLab with minimal effort. 1. Currently GitLab doesn't allow comments on arbitrary lines of code, so any Bitbucket comments out of bounds will be inserted as comments in the merge request. -1. Bitbucket Server allows multiple levels of threading. GitLab - import will collapse this into one discussion and quote part of the original - comment. +1. Bitbucket Server allows multiple levels of threading. GitLab import + will collapse this into one thread and quote part of the original comment. 1. Declined pull requests have unreachable commits, which prevents the GitLab importer from generating a proper diff. These pull requests will show up as empty changes. 1. Attachments in Markdown are currently not imported. 1. Task lists are not imported. 1. Emoji reactions are not imported +1. Project filtering does not support fuzzy search (only `starts with` or `full + match strings` are currently supported) ## How it works @@ -60,16 +61,16 @@ namespace that started the import process. 1. Sign in to GitLab and go to your dashboard. 1. Click on **New project**. 1. Click on the "Bitbucket Server" button. If the button is not present, enable the importer in - **Admin > Application Settings > Visibility and access controls > Import sources**. + **Admin > Application Settings > Visibility and access controls > Import sources**. -  +  1. Enter your Bitbucket Server credentials. -  +  1. Click on the projects that you'd like to import or **Import all projects**. - You can also select the namespace under which each project will be + You can also filter projects by name and select the namespace under which each project will be imported. -  +  diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md index 0afa32e4133..cf48189fa6e 100644 --- a/doc/user/project/import/gemnasium.md +++ b/doc/user/project/import/gemnasium.md @@ -5,13 +5,17 @@ instance or GitLab.com. ## Why is Gemnasium.com closed? -Gemnasium has been [acquired by GitLab](https://about.gitlab.com/press/releases/2018-01-30-gemnasium-acquisition.html) +Gemnasium was [acquired by GitLab](https://about.gitlab.com/press/releases/2018-01-30-gemnasium-acquisition.html) in January 2018. Since May 15, 2018, the services provided by Gemnasium are no longer available. The team behind Gemnasium has joined GitLab as the new Security Products team -and is working on a wider range of tools than just Dependency Scanning: -[SAST](../../application_security/sast/index.md), -[DAST](../../application_security/dast/index.md), -[Container Scanning](../../application_security/container_scanning/index.md) and more. +and is working on a [wide range of tools](../../application_security/index.md), +including: + +- [Dependency Scanning](../../application_security/dependency_scanning/index.md) +- [SAST](../../application_security/sast/index.md) +- [DAST](../../application_security/dast/index.md) +- [Container Scanning](../../application_security/container_scanning/index.md) + If you want to continue monitoring your dependencies, see the [Migrating to GitLab](#migrating-to-gitlab) section below. @@ -57,27 +61,27 @@ back to both GitLab and GitHub when completed. 1. Create a new project, and select the "CI/CD for external repo" tab: -  +  1. Use the "GitHub" button to connect your repositories. -  +  1. Select the project(s) to be set up with GitLab CI/CD and chose "Connect". -  +  - Once the configuration is done, you may click on your new - project on GitLab. + Once the configuration is done, you may click on your new + project on GitLab. -  +  - Your project is now mirrored on GitLab, where the Runners will be able to access - your source code and run your tests. + Your project is now mirrored on GitLab, where the Runners will be able to access + your source code and run your tests. - Optional step: If you set this up on GitLab.com, make sure the project is - public (in the project settings) if your GitHub project is public, since - the security feature is available only for [GitLab Ultimate](https://about.gitlab.com/pricing). + Optional step: If you set this up on GitLab.com, make sure the project is + public (in the project settings) if your GitHub project is public, since + the security feature is available only for [GitLab Ultimate](https://about.gitlab.com/pricing). 1. To set up the dependency scanning job, corresponding to what Gemnasium was doing, you must create a `.gitlab-ci.yml` file, or update it according to @@ -85,16 +89,16 @@ back to both GitLab and GitHub when completed. The mirroring is pull-only by default, so you may create or update the file on GitHub: -  +  1. Once your file has been committed, a new pipeline will be automatically triggered if your file is valid: -  +  1. The result of the job will be visible directly from the pipeline view: -  +  NOTE: **Note:** If you don't commit very often to your project, you may want to use diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index f48a158e2d3..2f87f257754 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -1,21 +1,21 @@ # Project importing from GitLab.com to your private GitLab instance -You can import your existing GitLab.com projects to your GitLab instance. But keep in mind that it is possible only if -GitLab.com integration is enabled on your GitLab instance. +You can import your existing GitLab.com projects to your GitLab instance, but keep in +mind that it is possible only if GitLab.com integration is enabled on your GitLab instance. [Read more about GitLab.com integration for self-managed GitLab instances](../../../integration/gitlab.md). To get to the importer page you need to go to "New project" page. >**Note:** -If you are interested in importing Wiki and Merge Request data to your new -instance, you'll need to follow the instructions for [project export](../settings/import_export.md) +If you are interested in importing Wiki and Merge Request data to your new instance, +you'll need to follow the instructions for [exporting a project](../settings/import_export.md#exporting-a-project-and-its-data) - + -Click on the "Import projects from GitLab.com" link and you will be redirected to GitLab.com +Go to the **Import Projects** tab, then click on **GitLab.com**, and you will be redirected to GitLab.com for permission to access your projects. After accepting, you'll be automatically redirected to the importer.  -To import a project, you can simple click "Import". The importer will import your repository and issues. +To import a project, click "Import". The importer will import your repository and issues. Once the importer is done, a new GitLab project will be created with your imported data. diff --git a/doc/user/project/import/img/bitbucket_server_import_select_project.png b/doc/user/project/import/img/bitbucket_server_import_select_project.png Binary files differdeleted file mode 100644 index e7fddef9955..00000000000 --- a/doc/user/project/import/img/bitbucket_server_import_select_project.png +++ /dev/null diff --git a/doc/user/project/import/img/bitbucket_server_import_select_project_v12_3.png b/doc/user/project/import/img/bitbucket_server_import_select_project_v12_3.png Binary files differnew file mode 100644 index 00000000000..1c344853cc8 --- /dev/null +++ b/doc/user/project/import/img/bitbucket_server_import_select_project_v12_3.png diff --git a/doc/user/project/import/img/gitlab_new_project_page.png b/doc/user/project/import/img/gitlab_new_project_page.png Binary files differdeleted file mode 100644 index c673724f436..00000000000 --- a/doc/user/project/import/img/gitlab_new_project_page.png +++ /dev/null diff --git a/doc/user/project/import/img/gitlab_new_project_page_v12_2.png b/doc/user/project/import/img/gitlab_new_project_page_v12_2.png Binary files differnew file mode 100644 index 00000000000..e79c27f32c0 --- /dev/null +++ b/doc/user/project/import/img/gitlab_new_project_page_v12_2.png diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 334be713aa5..ecd491d8e5b 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -10,7 +10,7 @@ 1. [From Gitea](gitea.md) 1. [From Perforce](perforce.md) 1. [From SVN](svn.md) -1. [From TFS](tfs.md) +1. [From TFVC](tfvc.md) 1. [From repo by URL](repo_by_url.md) 1. [By uploading a manifest file (AOSP)](manifest.md) 1. [From Gemnasium](gemnasium.md) @@ -34,7 +34,7 @@ This approach assumes all users from the self-hosted instance have already been If the users haven't been migrated yet, the user conducting the import will take the place of all references to the missing user(s). -If you need to migrate all data over, you can leverage our [api](../../../api/README.md) to migrate from self-hosted to GitLab.com. +If you need to migrate all data over, you can leverage our [api](../../../api/README.md) to migrate from self-hosted to GitLab.com. The order of assets to migrate from a self-hosted instance to GitLab is the following: 1. [Users](../../../api/users.md) @@ -54,5 +54,5 @@ perhaps from an old server to a new server for example, is to [back up the project](../../../raketasks/backup_restore.md), then restore it on the new server. -In the event of merging two GitLab instances together (for example, both instances have existing data on them and one can't be wiped), +In the event of merging two GitLab instances together (for example, both instances have existing data on them and one can't be wiped), refer to the instructions in [Migrating from self-hosted GitLab to GitLab.com](#migrating-from-self-hosted-gitlab-to-gitlabcom). diff --git a/doc/user/project/import/manifest.md b/doc/user/project/import/manifest.md index baf410d9c9e..161a4163e42 100644 --- a/doc/user/project/import/manifest.md +++ b/doc/user/project/import/manifest.md @@ -1,7 +1,6 @@ # Import multiple repositories by uploading a manifest file -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28811) in -GitLab 11.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28811) in GitLab 11.2. GitLab allows you to import all the required Git repositories based on a manifest file like the one used by the @@ -56,4 +55,4 @@ You can start the import with: to the import status page with projects list based on the manifest file. 1. Check the list and click **Import all repositories** to start the import. -  +  diff --git a/doc/user/project/import/phabricator.md b/doc/user/project/import/phabricator.md index 5c624e3aff6..b8f89caba24 100644 --- a/doc/user/project/import/phabricator.md +++ b/doc/user/project/import/phabricator.md @@ -15,6 +15,12 @@ Currently, only the following basic fields are imported: - Created at - Closed at +## Users + +The assignee and author of a user are deducted from a Task's owner and +author: If a user with the same username has access to the namespace +of the project being imported into, then the user will be linked. + ## Enabling this feature While this feature is incomplete, a feature flag is required to enable it so that diff --git a/doc/user/project/import/svn.md b/doc/user/project/import/svn.md index 7359487e1bf..d175ee87f26 100644 --- a/doc/user/project/import/svn.md +++ b/doc/user/project/import/svn.md @@ -9,13 +9,13 @@ between the two, for more information consult your favorite search engine. There are two approaches to SVN to Git migration: 1. [Git/SVN Mirror](#smooth-migration-with-a-gitsvn-mirror-using-subgit) which: - - Makes the GitLab repository to mirror the SVN project. - - Git and SVN repositories are kept in sync; you can use either one. - - Smoothens the migration process and allows to manage migration risks. + - Makes the GitLab repository to mirror the SVN project. + - Git and SVN repositories are kept in sync; you can use either one. + - Smoothens the migration process and allows to manage migration risks. 1. [Cut over migration](#cut-over-migration-with-svn2git) which: - - Translates and imports the existing data and history from SVN to Git. - - Is a fire and forget approach, good for smaller teams. + - Translates and imports the existing data and history from SVN to Git. + - Is a fire and forget approach, good for smaller teams. ## Smooth migration with a Git/SVN mirror using SubGit diff --git a/doc/user/project/import/tfs.md b/doc/user/project/import/tfs.md index b4597a4da60..7b3b11b9519 100644 --- a/doc/user/project/import/tfs.md +++ b/doc/user/project/import/tfs.md @@ -1,42 +1,5 @@ -# Migrating from TFS +--- +redirect_to: 'tfvc.md' +--- -[TFS](https://visualstudio.microsoft.com/tfs/) is a set of tools developed by Microsoft -which also includes a centralized version control system (TFVC) similar to Git. - -In this document, we emphasize on the TFVC to Git migration. - -## TFVC vs Git - -The following list illustrates the main differences between TFVC and Git: - -- **Git is distributed** whereas TFVC is centralized using a client-server - architecture. This translates to Git having a more flexible workflow since - your working area is a copy of the entire repository. This decreases the - overhead when switching branches or merging for example, since you don't have - to communicate with a remote server. -- **Storage method.** Changes in CVS are per file (changeset), while in Git - a committed file(s) is stored in its entirety (snapshot). That means that's - very easy in Git to revert or undo a whole change. - -Check also Microsoft's documentation on the -[comparison of Git and TFVC](https://docs.microsoft.com/en-us/azure/devops/repos/tfvc/comparison-git-tfvc?view=azure-devops) -and the Wikipedia -[comparison of version control software](https://en.wikipedia.org/wiki/Comparison_of_version_control_software). - -## Why migrate - -Migrating to Git/GitLab there is: - -- **No licensing costs**, Git is GPL while TFVC is proprietary. -- **Shorter learning curve**, Git has a big community and a vast number of - tutorials to get you started (see our [Git topic](../../../topics/git/index.md)). -- **Integration with modern tools**, migrating to Git and GitLab you can have - an open source end-to-end software development platform with built-in version - control, issue tracking, code review, CI/CD, and more. - -## How to migrate - -The best option to migrate from TFVC to Git is to use the -[`git-tfs`](https://github.com/git-tfs/git-tfs) tool. A specific guide for the -migration exists: -[Migrate TFS to Git](https://github.com/git-tfs/git-tfs/blob/master/doc/usecases/migrate_tfs_to_git.md). +This document was moved to [another location](tfvc.md). diff --git a/doc/user/project/import/tfvc.md b/doc/user/project/import/tfvc.md new file mode 100644 index 00000000000..375522b77d0 --- /dev/null +++ b/doc/user/project/import/tfvc.md @@ -0,0 +1,46 @@ +--- +type: concepts +--- + +# Migrating from TFVC to Git + +Team Foundation Server (TFS), renamed [Azure DevOps Server](https://azure.microsoft.com/en-us/services/devops/server/) +in 2019, is a set of tools developed by Microsoft which also includes +[Team Foundation Version Control](https://docs.microsoft.com/en-us/azure/devops/repos/tfvc/overview) +(TFVC), a centralized version control system similar to Git. + +In this document, we focus on the TFVC to Git migration. + +## TFVC vs Git + +The main differences between TFVC and Git are: + +- **Git is distributed:** While TFVC is centralized using a client-server architecture, + Git is distributed. This translates to Git having a more flexible workflow since + you work with a copy of the entire repository. This allows you to quickly + switch branches or merge, for example, without needing to communicate with a remote server. +- **Storage:** Changes in a centralized version control system are per file (changeset), + while in Git a committed file is stored in its entirety (snapshot). That means that it is + very easy to revert or undo a whole change in Git. + +For more information, see: + +- Microsoft's [comparison of Git and TFVC](https://docs.microsoft.com/en-us/azure/devops/repos/tfvc/comparison-git-tfvc?view=azure-devops). +- The Wikipedia [comparison of version control software](https://en.wikipedia.org/wiki/Comparison_of_version_control_software). + +## Why migrate + +Advantages of migrating to Git/GitLab: + +- **No licensing costs:** Git is open source, while TFVC is proprietary. +- **Shorter learning curve:** Git has a big community and a vast number of + tutorials to get you started (see our [Git topic](../../../topics/git/index.md)). +- **Integration with modern tools:** After migrating to Git and GitLab, you will have + an open source, end-to-end software development platform with built-in version + control, issue tracking, code review, CI/CD, and more. + +## How to migrate + +The best option to migrate from TFVC to Git is to use the [`git-tfs`](https://github.com/git-tfs/git-tfs) +tool. Read the [Migrate TFS to Git](https://github.com/git-tfs/git-tfs/blob/master/doc/usecases/migrate_tfs_to_git.md) +guide for more details. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index f332281fa82..64c4066683b 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -17,7 +17,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow - - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **(STARTER)** + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards): Allow your teams to create their own workflows (Issue Boards) for the same project - [Repositories](repository/index.md): Host your code in a fully integrated platform - [Branches](repository/branches/index.md): use Git branching strategies to @@ -34,7 +34,7 @@ When you create a project in GitLab, you'll have access to a large number of - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow - - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **(STARTER)** + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards): Allow your teams to create their own workflows (Issue Boards) for the same project - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - [Merge Request Approvals](merge_requests/merge_request_approvals.md): Ask for approval before @@ -52,6 +52,9 @@ When you create a project in GitLab, you'll have access to a large number of templates for issue and merge request description fields for your project - [Slash commands (quick actions)](quick_actions.md): Textual shortcuts for common actions on issues or merge requests +- [Autocomplete characters](autocomplete_characters.md): Autocomplete + references to users, groups, issues, merge requests, and other GitLab + elements. - [Web IDE](web_ide/index.md) **GitLab CI/CD:** @@ -63,15 +66,15 @@ When you create a project in GitLab, you'll have access to a large number of to automatically set up your app's deployment - [Enable and disable GitLab CI](../../ci/enable_or_disable_ci.md) - [Pipelines](../../ci/pipelines.md): Configure and visualize - your GitLab CI/CD pipelines from the UI - - [Scheduled Pipelines](pipelines/schedules.md): Schedule a pipeline - to start at a chosen time - - [Pipeline Graphs](../../ci/pipelines.md#visualizing-pipelines): View your - entire pipeline from the UI - - [Job artifacts](pipelines/job_artifacts.md): Define, - browse, and download job artifacts - - [Pipeline settings](pipelines/settings.md): Set up Git strategy (choose the default way your repository is fetched from GitLab in a job), - timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more + your GitLab CI/CD pipelines from the UI + - [Scheduled Pipelines](pipelines/schedules.md): Schedule a pipeline + to start at a chosen time + - [Pipeline Graphs](../../ci/pipelines.md#visualizing-pipelines): View your + entire pipeline from the UI + - [Job artifacts](pipelines/job_artifacts.md): Define, + browse, and download job artifacts + - [Pipeline settings](pipelines/settings.md): Set up Git strategy (choose the default way your repository is fetched from GitLab in a job), + timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more - [Kubernetes cluster integration](clusters/index.md): Connecting your GitLab project with a Kubernetes cluster - [Feature Flags](operations/feature_flags.md): Feature flags allow you to ship a project in @@ -95,7 +98,8 @@ When you create a project in GitLab, you'll have access to a large number of - [Maven packages](packages/maven_repository.md): your private Maven repository in GitLab. **(PREMIUM)** - [NPM packages](packages/npm_registry.md): your private NPM package registry in GitLab. **(PREMIUM)** - [Code owners](code_owners.md): specify code owners for certain files **(STARTER)** -- [License Management](../application_security/license_management/index.md): approve and blacklist licenses for projects. **(ULTIMATE)** +- [License Compliance](../application_security/license_management/index.md): approve and blacklist licenses for projects. **(ULTIMATE)** +- [Dependency List](../application_security/dependency_list/index.md): view project dependencies. **(ULTIMATE)** ### Project integrations @@ -221,7 +225,7 @@ There are numerous [APIs](../../api/README.md) to use with your projects: - [Badges](../../api/project_badges.md) - [Clusters](../../api/project_clusters.md) -- [Discussions](../../api/discussions.md) +- [Threads](../../api/discussions.md) - [General](../../api/projects.md) - [Import/export](../../api/project_import_export.md) - [Issue Board](../../api/boards.md) diff --git a/doc/user/project/insights/img/insights_example_bar_chart.png b/doc/user/project/insights/img/insights_example_bar_chart.png Binary files differindex eb96eb4a90f..5fd24265b32 100644 --- a/doc/user/project/insights/img/insights_example_bar_chart.png +++ b/doc/user/project/insights/img/insights_example_bar_chart.png diff --git a/doc/user/project/insights/img/insights_example_bar_time_series_chart.png b/doc/user/project/insights/img/insights_example_bar_time_series_chart.png Binary files differindex 28aa81939d8..2e32df3e2b1 100644 --- a/doc/user/project/insights/img/insights_example_bar_time_series_chart.png +++ b/doc/user/project/insights/img/insights_example_bar_time_series_chart.png diff --git a/doc/user/project/insights/img/insights_example_line_chart.png b/doc/user/project/insights/img/insights_example_line_chart.png Binary files differindex 131dbedd35e..9460b104339 100644 --- a/doc/user/project/insights/img/insights_example_line_chart.png +++ b/doc/user/project/insights/img/insights_example_line_chart.png diff --git a/doc/user/project/insights/img/insights_example_pie_chart.png b/doc/user/project/insights/img/insights_example_pie_chart.png Binary files differindex 518ed7338f9..3480bce6738 100644 --- a/doc/user/project/insights/img/insights_example_pie_chart.png +++ b/doc/user/project/insights/img/insights_example_pie_chart.png diff --git a/doc/user/project/insights/img/insights_example_stacked_bar_chart.png b/doc/user/project/insights/img/insights_example_stacked_bar_chart.png Binary files differindex aafec4b394e..a3fb50488f5 100644 --- a/doc/user/project/insights/img/insights_example_stacked_bar_chart.png +++ b/doc/user/project/insights/img/insights_example_stacked_bar_chart.png diff --git a/doc/user/project/insights/img/insights_sidebar_link.png b/doc/user/project/insights/img/insights_sidebar_link.png Binary files differindex aadb5745992..9fc449baf68 100644 --- a/doc/user/project/insights/img/insights_sidebar_link.png +++ b/doc/user/project/insights/img/insights_sidebar_link.png diff --git a/doc/user/project/insights/img/project_insights.png b/doc/user/project/insights/img/project_insights.png Binary files differindex 2d0292dda54..83674c94110 100644 --- a/doc/user/project/insights/img/project_insights.png +++ b/doc/user/project/insights/img/project_insights.png diff --git a/doc/user/project/integrations/custom_issue_tracker.md b/doc/user/project/integrations/custom_issue_tracker.md index 23f1ce7a15a..7c7263704f9 100644 --- a/doc/user/project/integrations/custom_issue_tracker.md +++ b/doc/user/project/integrations/custom_issue_tracker.md @@ -17,6 +17,6 @@ Once you have configured and enabled Custom Issue Tracker Service you'll see a l ## Referencing issues -- Issues are referenced with `ANYTHING-<ID>`, where `ANYTHING` can be any string and `<ID>` is a number used in the target project of the custom integration (example `PROJECT-143`). +- Issues are referenced with `ANYTHING-<ID>`, where `ANYTHING` can be any string and `<ID>` is a number used in the target project of the custom integration (example `PROJECT-143`). - `ANYTHING` is a placeholder to differentiate against GitLab issues, which are referenced with `#<ID>`. You can use a project name or project key to replace it for example. -- So with the example above, `PROJECT-143` would refer to `https://customissuetracker.com/project-name/143`.
\ No newline at end of file +- So with the example above, `PROJECT-143` would refer to `https://customissuetracker.com/project-name/143`. diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md index 8e062ca627b..91de64be1fa 100644 --- a/doc/user/project/integrations/gitlab_slack_application.md +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -1,4 +1,4 @@ -# GitLab Slack application **[FREE ONLY]** +# GitLab Slack application **(FREE ONLY)** NOTE: **Note:** The GitLab Slack application is only configurable for GitLab.com. It will **not** diff --git a/doc/user/project/integrations/img/download_as_csv.png b/doc/user/project/integrations/img/download_as_csv.png Binary files differnew file mode 100644 index 00000000000..0ed5ab8db89 --- /dev/null +++ b/doc/user/project/integrations/img/download_as_csv.png diff --git a/doc/user/project/integrations/img/embed_metrics.png b/doc/user/project/integrations/img/embed_metrics.png Binary files differnew file mode 100644 index 00000000000..6f9660c9aec --- /dev/null +++ b/doc/user/project/integrations/img/embed_metrics.png diff --git a/doc/user/project/integrations/img/generate_link_to_chart.png b/doc/user/project/integrations/img/generate_link_to_chart.png Binary files differnew file mode 100644 index 00000000000..03e018969b1 --- /dev/null +++ b/doc/user/project/integrations/img/generate_link_to_chart.png diff --git a/doc/user/project/integrations/img/github_configuration.png b/doc/user/project/integrations/img/github_configuration.png Binary files differindex 9f2d47921c7..5798b826681 100644 --- a/doc/user/project/integrations/img/github_configuration.png +++ b/doc/user/project/integrations/img/github_configuration.png diff --git a/doc/user/project/integrations/img/jira_add_user_to_group.png b/doc/user/project/integrations/img/jira_add_user_to_group.png Binary files differindex d8cf541a81e..b63a851a987 100644 --- a/doc/user/project/integrations/img/jira_add_user_to_group.png +++ b/doc/user/project/integrations/img/jira_add_user_to_group.png diff --git a/doc/user/project/integrations/img/jira_added_user_to_group.png b/doc/user/project/integrations/img/jira_added_user_to_group.png Binary files differindex b3e29a65d6e..f5120a8d42e 100644 --- a/doc/user/project/integrations/img/jira_added_user_to_group.png +++ b/doc/user/project/integrations/img/jira_added_user_to_group.png diff --git a/doc/user/project/integrations/img/jira_api_token.png b/doc/user/project/integrations/img/jira_api_token.png Binary files differindex 29689271bf7..d9d37713a4d 100644 --- a/doc/user/project/integrations/img/jira_api_token.png +++ b/doc/user/project/integrations/img/jira_api_token.png diff --git a/doc/user/project/integrations/img/jira_api_token_menu.png b/doc/user/project/integrations/img/jira_api_token_menu.png Binary files differindex 1aca1d78f36..a10a59a243d 100644 --- a/doc/user/project/integrations/img/jira_api_token_menu.png +++ b/doc/user/project/integrations/img/jira_api_token_menu.png diff --git a/doc/user/project/integrations/img/jira_create_new_group.png b/doc/user/project/integrations/img/jira_create_new_group.png Binary files differindex 84be3a94a45..4ab7a5eae4e 100644 --- a/doc/user/project/integrations/img/jira_create_new_group.png +++ b/doc/user/project/integrations/img/jira_create_new_group.png diff --git a/doc/user/project/integrations/img/jira_create_new_user.png b/doc/user/project/integrations/img/jira_create_new_user.png Binary files differindex 8460dc98ef9..c74933298e3 100644 --- a/doc/user/project/integrations/img/jira_create_new_user.png +++ b/doc/user/project/integrations/img/jira_create_new_user.png diff --git a/doc/user/project/integrations/img/jira_group_access.png b/doc/user/project/integrations/img/jira_group_access.png Binary files differindex 58cf114bd55..e33f2eed242 100644 --- a/doc/user/project/integrations/img/jira_group_access.png +++ b/doc/user/project/integrations/img/jira_group_access.png diff --git a/doc/user/project/integrations/img/jira_issue_reference.png b/doc/user/project/integrations/img/jira_issue_reference.png Binary files differindex a3e80c1b054..db8bc4f0bb9 100644 --- a/doc/user/project/integrations/img/jira_issue_reference.png +++ b/doc/user/project/integrations/img/jira_issue_reference.png diff --git a/doc/user/project/integrations/img/jira_merge_request_close.png b/doc/user/project/integrations/img/jira_merge_request_close.png Binary files differindex 1c089c94207..9a176daf5f4 100644 --- a/doc/user/project/integrations/img/jira_merge_request_close.png +++ b/doc/user/project/integrations/img/jira_merge_request_close.png diff --git a/doc/user/project/integrations/img/jira_service_page.png b/doc/user/project/integrations/img/jira_service_page.png Binary files differdeleted file mode 100644 index 80dd65ea24e..00000000000 --- a/doc/user/project/integrations/img/jira_service_page.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_service_page_v12_2.png b/doc/user/project/integrations/img/jira_service_page_v12_2.png Binary files differnew file mode 100644 index 00000000000..ba7dad9b438 --- /dev/null +++ b/doc/user/project/integrations/img/jira_service_page_v12_2.png diff --git a/doc/user/project/integrations/img/jira_user_management_link.png b/doc/user/project/integrations/img/jira_user_management_link.png Binary files differindex 43ef18da6c8..caecd1d71fc 100644 --- a/doc/user/project/integrations/img/jira_user_management_link.png +++ b/doc/user/project/integrations/img/jira_user_management_link.png diff --git a/doc/user/project/integrations/img/mattermost_configuration.png b/doc/user/project/integrations/img/mattermost_configuration.png Binary files differindex 75ef0310f2d..6abf5c5c8d6 100644 --- a/doc/user/project/integrations/img/mattermost_configuration.png +++ b/doc/user/project/integrations/img/mattermost_configuration.png diff --git a/doc/user/project/integrations/img/prometheus_dashboard_area_panel_type.png b/doc/user/project/integrations/img/prometheus_dashboard_area_panel_type.png Binary files differnew file mode 100644 index 00000000000..9fdb6dfddac --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_dashboard_area_panel_type.png diff --git a/doc/user/project/integrations/img/prometheus_dashboard_single_stat_panel_type.png b/doc/user/project/integrations/img/prometheus_dashboard_single_stat_panel_type.png Binary files differnew file mode 100644 index 00000000000..2d7dfb27b49 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_dashboard_single_stat_panel_type.png diff --git a/doc/user/project/integrations/img/prometheus_service_alerts.png b/doc/user/project/integrations/img/prometheus_service_alerts.png Binary files differindex a81dfe7da14..609c5e5196c 100644 --- a/doc/user/project/integrations/img/prometheus_service_alerts.png +++ b/doc/user/project/integrations/img/prometheus_service_alerts.png diff --git a/doc/user/project/integrations/img/slack_configuration.png b/doc/user/project/integrations/img/slack_configuration.png Binary files differindex a14d2969488..10d2cda6dc7 100644 --- a/doc/user/project/integrations/img/slack_configuration.png +++ b/doc/user/project/integrations/img/slack_configuration.png diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index ca990ee6c32..61f6f6c9412 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -93,7 +93,7 @@ even if the status you are changing to is the same. After saving the configuration, your GitLab project will be able to interact with all Jira projects in your Jira instance and you'll see the Jira link on the GitLab project pages that takes you to the appropriate Jira project. - + ## Jira issues diff --git a/doc/user/project/integrations/jira_cloud_configuration.md b/doc/user/project/integrations/jira_cloud_configuration.md index 5a5ba2dd168..1d5a4a3d4c7 100644 --- a/doc/user/project/integrations/jira_cloud_configuration.md +++ b/doc/user/project/integrations/jira_cloud_configuration.md @@ -9,7 +9,7 @@ below to create one: It is important that the user associated with this email address has *write* access to projects in Jira. -2. Click **Create API token**. +1. Click **Create API token**.  diff --git a/doc/user/project/integrations/jira_server_configuration.md b/doc/user/project/integrations/jira_server_configuration.md index 32991714973..1efd0ff3944 100644 --- a/doc/user/project/integrations/jira_server_configuration.md +++ b/doc/user/project/integrations/jira_server_configuration.md @@ -3,8 +3,8 @@ We need to create a user in Jira which will have access to all projects that need to integrate with GitLab. -As an example, we'll create a user named `gitlab` and add it to the `Jira-developers` -group. +As an example, we'll create a user named `gitlab` and add it to a new group +named `gitlab-developers`. NOTE: **Note** It is important that the Jira user created for the integration is given 'write' @@ -13,46 +13,46 @@ access to your Jira projects. This is covered in the process below. 1. Log in to your Jira instance as an administrator and under **Jira Administration** go to **User Management** to create a new user. -  +  1. The next step is to create a new user (e.g., `gitlab`) who has write access to projects in Jira. Enter the user's name and a _valid_ e-mail address since Jira sends a verification e-mail to set up the password. _**Note:** Jira creates the username automatically by using the e-mail prefix. You can change it later, if needed. Our integration does not support SSO (such as SAML). You will need to create - an HTTP basic authentication password. You can do this by visiting the user - profile, looking up the username, and setting a password._ + an HTTP basic authentication password. You can do this by visiting the user + profile, looking up the username, and setting a password._ -  +  1. Create a `gitlab-developers` group. (We will give this group write access to Jira projects in a later step). Go to the **Groups** tab on the left, and select **Add group**. -  +  - Give it a name and click **Add group**. + Give it a name and click **Add group**. 1. Add the `gitlab` user to the `gitlab-developers` group by clicking **Edit members**. The `gitlab-developers` group should be listed in the leftmost box as a selected group. Under **Add members to selected group(s)**, enter `gitlab`. -  - +  + Click **Add selected users** and `gitlab` should appear in the **Group member(s)** box. This membership is saved automatically. - -  + +  1. To give the newly-created group 'write' access, you need to create a **Permission Scheme**. To do this, click the gear icon and select **Issues**. Then click **Permission Schemes**. Click **Add Permission Scheme** and enter a **Name** and, optionally, a **Description**. - + 1. Once your permission scheme is created, you'll be taken back to the permissions scheme list. - Locate your new permissions scheme and click **Permissions**. Next to **Administer Projects**, + Locate your new permissions scheme and click **Permissions**. Next to **Administer Projects**, click **Edit**. In the resulting dialog box, select **Group** and select `gitlab-developers` from the dropdown. -  +  The Jira configuration is complete. Write down the new Jira username and its password as they will be needed when [configuring GitLab in the next section](jira.md#configuring-gitlab). diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index ea58a08e127..6e0f39956d3 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -14,7 +14,7 @@ To enable Mattermost integration you must create an incoming webhook integration 1. Save it, copy the **Webhook URL**, we'll need this later for GitLab. There might be some cases that Incoming Webhooks are blocked by admin, ask your mattermost admin to enable -it on `https://mattermost.example/admin_console/integrations/custom`. +it on **Mattermost System Console > Integrations > Integration Management**, or on **Mattermost System Console > Integrations > Custom Integrations** in Mattermost versions 5.11 and earlier. Display name override is not enabled by default, you need to ask your admin to enable it on that same section. diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index 41be26c1d30..a3a2568445e 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -40,17 +40,13 @@ the administrator console. 1. Log in with an account that has admin privileges and navigate to the system console. -  - - --- +  1. Click **Custom integrations** and set **Enable Custom Slash Commands**, **Enable custom integrations to override usernames**, and **Override custom integrations to override profile picture icons** to true -  - - --- +  1. Click **Save** at the bottom to save the changes. @@ -62,14 +58,12 @@ the administrator console. A screen will appear with all the values you need to copy in Mattermost as described in the next step. Leave the window open. - >**Note:** - GitLab will propose some values for the Mattermost settings. The only one - required to copy-paste as-is is the **Request URL**, all the others are just - suggestions. - -  + NOTE: **Note:** + GitLab will propose some values for the Mattermost settings. The only one + required to copy-paste as-is is the **Request URL**, all the others are just + suggestions. - --- +  1. Proceed to the next step and create a slash command in Mattermost with the above values. @@ -83,44 +77,38 @@ in a new slash command. 1. Back to Mattermost, under your team page settings, you should see the **Integrations** option. -  - - --- +  1. Go to the **Slash Commands** integration and add a new one by clicking the **Add Slash Command** button. -  - - --- +  1. Fill in the options for the custom command as described in [step 2](#step-2-open-the-mattermost-slash-commands-service-in-gitlab). - >**Note:** - If you plan on connecting multiple projects, pick a slash command trigger - word that relates to your projects such as `/gitlab-project-name` or even - just `/project-name`. Only use `/gitlab` if you will only connect a single - project to your Mattermost team. + NOTE: **Note:** + If you plan on connecting multiple projects, pick a slash command trigger + word that relates to your projects such as `/gitlab-project-name` or even + just `/project-name`. Only use `/gitlab` if you will only connect a single + project to your Mattermost team. -  +  1. After you set up all the values, copy the token (we will use it below) and click **Done**. -  +  ### Step 4. Copy the Mattermost token into the Mattermost slash command service 1. In GitLab, paste the Mattermost token you copied in the previous step and check the **Active** checkbox. -  +  1. Click **Save changes** for the changes to take effect. ---- - You are now set to start using slash commands in Mattermost that talk to the GitLab project you configured. diff --git a/doc/user/project/integrations/mock_ci.md b/doc/user/project/integrations/mock_ci.md index 1c64b275d6e..886094a6531 100644 --- a/doc/user/project/integrations/mock_ci.md +++ b/doc/user/project/integrations/mock_ci.md @@ -5,9 +5,9 @@ To set up the mock CI service server, respond to the following endpoints - `commit_status`: `#{project.namespace.path}/#{project.path}/status/#{sha}.json` - - Have your service return `200 { status: ['failed'|'canceled'|'running'|'pending'|'success'|'success-with-warnings'|'skipped'|'not_found'] }` + - Have your service return `200 { status: ['failed'|'canceled'|'running'|'pending'|'success'|'success-with-warnings'|'skipped'|'not_found'] }` - If the service returns a 404, it is interpreted as `pending` - `build_page`: `#{project.namespace.path}/#{project.path}/status/#{sha}` - - Just where the build is linked to, doesn't matter if implemented + - Just where the build is linked to, doesn't matter if implemented For an example of a mock CI server, see [`gitlab-org/gitlab-mock-ci-service`](https://gitlab.com/gitlab-org/gitlab-mock-ci-service) diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index 62e08a183f7..67274e8d924 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -20,10 +20,10 @@ Below, you will find a list of the currently supported ones accompanied with com Click on the service links to see further configuration instructions and details. -| Service | Description | +| Service | Description | | ------- | ----------- | -| Asana | Asana - Teamwork without email | -| Assembla | Project Management Software (Source Commits Endpoint) | +| Asana | Asana - Teamwork without email | +| Assembla | Project Management Software (Source Commits Endpoint) | | [Atlassian Bamboo CI](bamboo.md) | A continuous integration and build server | | Buildkite | Continuous integration and deployments | | [Bugzilla](bugzilla.md) | Bugzilla issue tracker | @@ -48,7 +48,7 @@ Click on the service links to see further configuration instructions and details | Pipelines emails | Email the pipeline status to a list of recipients | | [Slack Notifications](slack.md) | Send GitLab events (e.g. issue created) to Slack as notifications | | [Slack slash commands](slack_slash_commands.md) **(CORE ONLY)** | Use slash commands in Slack to control GitLab | -| [GitLab Slack application](gitlab_slack_application.md) **[FREE ONLY]** | Use Slack's official application | +| [GitLab Slack application](gitlab_slack_application.md) **(FREE ONLY)** | Use Slack's official application | | PivotalTracker | Project Management Software (Source Commits Endpoint) | | [Prometheus](prometheus.md) | Monitor the performance of your deployed apps | | Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop | diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 01b6650bfab..a80332adc46 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -98,7 +98,10 @@ You can view the performance dashboard for an environment by [clicking on the mo > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3799) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.6. -Additional metrics can be monitored by adding them on the Prometheus integration page. Once saved, they will be displayed on the environment performance dashboard. +Custom metrics can be monitored by adding them on the Prometheus integration page. Once saved, they will be displayed on the environment performance dashboard provided that either: + +- A [connected Kubernetes cluster](../clusters/index.md#adding-and-removing-clusters) with the environment scope of `*` is used and [Prometheus installed on the cluster](#enabling-prometheus-integration), or +- Prometheus is [manually configured](#manual-configuration-of-prometheus).  @@ -121,6 +124,157 @@ GitLab supports a limited set of [CI variables](../../../ci/variables/README.htm To specify a variable in a query, enclose it in curly braces with a leading percent. For example: `%{ci_environment_slug}`. +### Defining custom dashboards per project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/59974) in GitLab 12.1. + +By default, all projects include a GitLab-defined Prometheus dashboard, which +includes a few key metrics, but you can also define your own custom dashboards. + +NOTE: **Note:** +The custom metrics as defined below do not support alerts, unlike +[additional metrics](#adding-additional-metrics-premium). + +Dashboards have several components: + +- Panel groups, which comprise panels. +- Panels, which support one or more metrics. + +To configure a custom dashboard: + +1. Create a YAML file with the `.yml` extension under your repository's root + directory inside `.gitlab/dashboards/`. For example, create + `.gitlab/dashboards/prom_alerts.yml` with the following contents: + + ```yaml + dashboard: 'Dashboard Title' + panel_groups: + - group: 'Group Title' + panels: + - type: area-chart + title: "Chart Title" + y_label: "Y-Axis" + metrics: + - id: metric_of_ages + query_range: 'http_requests_total' + label: "Metric of Ages" + unit: "count" + ``` + + The above sample dashboard would display a single area chart. Each file should + define the layout of the dashboard and the Prometheus queries used to populate + data. + +1. Save the file, commit, and push to your repository. +1. Navigate to your project's **Operations > Metrics** and choose the custom + dashboard from the dropdown. + +NOTE: **Note:** +Configuration files nested under subdirectories of `.gitlab/dashboards` are not +supported and will not be available in the UI. + +The following tables outline the details of expected properties. + +**Dashboard properties:** + +| Property | Type | Required | Description | +| ------ | ------ | ------ | ------ | +| `dashboard` | string | yes | Heading for the dashboard. Only one dashboard should be defined per file. | +| `panel_groups` | array | yes | The panel groups which should be on the dashboard. | + +**Panel group (`panel_groups`) properties:** + +| Property | Type | Required | Description | +| ------ | ------ | ------ | ------ | +| `group` | string | required | Heading for the panel group. | +| `priority` | number | optional, defaults to order in file | Order to appear on the dashboard. Higher number means higher priority, which will be higher on the page. Numbers do not need to be consecutive. | +| `panels` | array | required | The panels which should be in the panel group. | + +**Panel (`panels`) properties:** + +| Property | Type | Required | Description | +| ------ | ------ | ------ | ------- | +| `type` | enum | no, defaults to `area-chart` | Specifies the chart type to use, can be `area-chart` or `line-chart` | +| `title` | string | yes | Heading for the panel. | +| `y_label` | string | no, but highly encouraged | Y-Axis label for the panel. | +| `weight` | number | no, defaults to order in file | Order to appear within the grouping. Lower number means higher priority, which will be higher on the page. Numbers do not need to be consecutive. | +| `metrics` | array | yes | The metrics which should be displayed in the panel. | + +**Metrics (`metrics`) properties:** + +| Property | Type | Required | Description | +| ------ | ------ | ------ | ------ | +| `id` | string | no | Used for associating dashboard metrics with database records. Must be unique across dashboard configuration files. Required for [alerting](#setting-up-alerts-for-prometheus-metrics-ultimate) (support not yet enabled, see [relevant issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/60319)). | +| `unit` | string | yes | Defines the unit of the query's return data. | +| `label` | string | no, but highly encouraged | Defines the legend-label for the query. Should be unique within the panel's metrics. | +| `query` | string | yes if `query_range` is not defined | Defines the Prometheus query to be used to populate the chart/panel. If defined, the `query` endpoint of the [Prometheus API](https://prometheus.io/docs/prometheus/latest/querying/api/) will be utilized. | +| `query_range` | string | yes if `query` is not defined | Defines the Prometheus query to be used to populate the chart/panel. If defined, the `query_range` endpoint of the [Prometheus API](https://prometheus.io/docs/prometheus/latest/querying/api/) will be utilized. | + +#### Panel types for dashboards + +The below panel types are supported in monitoring dashboards. + +##### Area + +To add an area panel type to a dashboard, look at the following sample dashboard file: + +```yaml +dashboard: 'Dashboard Title' +panel_groups: + - group: 'Group Title' + panels: + - type: area-chart + title: "Chart Title" + y_label: "Y-Axis" + metrics: + - id: 10 + query_range: 'http_requests_total' + label: "Metric of Ages" + unit: "count" +``` + +Note the following properties: + +| Property | Type | Required | Description | +| ------ | ------ | ------ | ------ | +| type | string | no | Type of panel to be rendered. Optional for area panel types | +| query_range | yes | required | For area panel types, you must use a [range query](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries) | + + + +##### Single Stat + +To add a single stat panel type to a dashboard, look at the following sample dashboard file: + +```yaml +dashboard: 'Dashboard Title' +panel_groups: + - group: 'Group Title' + panels: + - title: "Single Stat" + type: "single-stat" + metrics: + - id: 10 + query: 'max(go_memstats_alloc_bytes{job="prometheus"})' + unit: MB + label: "Total" +``` + +Note the following properties: + +| Property | Type | Required | Description | +| ------ | ------ | ------ | ------ | +| type | string | yes | Type of panel to be rendered. For single stat panel types, set to `single-stat` | +| query | string | yes | For single stat panel types, you must use an [instant query](https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries) | + + + +### Downloading data as CSV + +Data from Prometheus charts on the metrics dashboard can be downloaded as CSV. + + + ### Setting up alerts for Prometheus metrics **(ULTIMATE)** #### Managed Prometheus instances @@ -169,7 +323,20 @@ Alerts can be used to trigger actions, like open an issue automatically (enabled 1. Optionally, select whether to send an email notification to the developers of the project. 1. Click **Save changes**. -Once enabled, an issue will be opened automatically when an alert is triggered. The author of the issue will be the GitLab Alert Bot. To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template. +Once enabled, an issue will be opened automatically when an alert is triggered which contains values extracted from [alert's payload](https://prometheus.io/docs/alerting/configuration/#webhook_config +): + +- Issue author: `GitLab Alert Bot` +- Issue title: Extract from `annotations/title`, `annotations/summary` or `labels/alertname` +- Alert `Summary`: A list of properties + - `starts_at`: Alert start time via `startsAt` + - `full_query`: Alert query extracted from `generatorURL` + - Optional list of attached annotations extracted from `annotations/*` +- Alert [GFM](../../markdown.md): GitLab Flavored Markdown from `annotations/gitlab_incident_markdown` + +To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template, which will apply to all incidents. To limit quick actions or other information to only specific types of alerts, use the `annotations/gitlab_incident_markdown` field. + +Since [version 12.2](https://gitlab.com/gitlab-org/gitlab-ce/issues/63373), GitLab will tag each incident issue with the `incident` label automatically. If the label does not yet exist, it will be created automatically as well. If the metric exceeds the threshold of the alert for over 5 minutes, an email will be sent to all [Maintainers and Owners](../../permissions.md#project-members-permissions) of the project. @@ -193,6 +360,31 @@ Prometheus server.  +## Embedding metric charts within GitLab Flavored Markdown + +> [Introduced][ce-29691] in GitLab 12.2. +> Requires [Kubernetes](prometheus_library/kubernetes.md) metrics. + +It is possible to display metrics charts within [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). + +To display a metric chart, include a link of the form `https://<root_url>/<project>/environments/<environment_id>/metrics`. + +A single chart may also be embedded. You can generate a link to the chart via the dropdown located on the right side of the chart: + + + +The following requirements must be met for the metric to unfurl: + +- The `<environment_id>` must correspond to a real environment. +- Prometheus must be monitoring the environment. +- The GitLab instance must be configured to receive data from the environment. +- The user must be allowed access to the monitoring dashboard for the environment ([Reporter or higher](../../permissions.md)). +- The dashboard must have data within the last 8 hours. + + If all of the above are true, then the metric will unfurl as seen below: + + + ## Troubleshooting If the "No data found" screen continues to appear, it could be due to: @@ -215,4 +407,5 @@ If the "No data found" screen continues to appear, it could be due to: [ci-environment-slug]: ../../../ci/variables/#predefined-environment-variables [ce-8935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935 [ce-10408]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10408 +[ce-29691]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29691 [promgldocs]: ../../../administration/monitoring/prometheus/index.md diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md index 6be8fc82431..90a725f271b 100644 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -1,4 +1,5 @@ # Monitoring HAProxy + > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12621) in GitLab 9.4 GitLab has support for automatically detecting and monitoring HAProxy. This is provided by leveraging the [HAProxy Exporter](https://github.com/prometheus/haproxy_exporter), which translates HAProxy statistics into a Prometheus readable form. diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 0d300d3c418..9e17edf7d36 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -13,15 +13,15 @@ integration services must be enabled. - Average Memory Usage (MB): - ``` - avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 - ``` + ``` + avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 + ``` - Average CPU Utilization (%): - ``` - avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) - ``` + ``` + avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-([^c].*|c([^a]|a([^n]|n([^a]|a([^r]|r[^y])))).*|)-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) + ``` ## Configuring Prometheus to monitor for Kubernetes metrics @@ -48,12 +48,12 @@ These metrics expect the [Deployment](https://kubernetes.io/docs/concepts/worklo - Average Memory Usage (MB) - ``` - avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 - ``` + ``` + avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 + ``` - Average CPU Utilization (%) - ``` - avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) - ``` + ``` + avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) + ``` diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index bac7eecfce4..25b000b2753 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -5,18 +5,18 @@ the **Redmine** service, and fill in the required details on the page as described in the table below. - | Field | Description | - | ----- | ----------- | - | `description` | A name for the issue tracker (to differentiate between instances, for example) | - | `project_url` | The URL to the project in Redmine which is being linked to this GitLab project | - | `issues_url` | The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | - | `new_issue_url` | This is the URL to create a new issue in Redmine for the project linked to this GitLab project. **This is currently not being used and will be removed in a future release.** | + | Field | Description | + | ----- | ----------- | + | `description` | A name for the issue tracker (to differentiate between instances, for example) | + | `project_url` | The URL to the project in Redmine which is being linked to this GitLab project | + | `issues_url` | The URL to the issue in Redmine project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | + | `new_issue_url` | This is the URL to create a new issue in Redmine for the project linked to this GitLab project. **This is currently not being used and will be removed in a future release.** | - Once you have configured and enabled Redmine you'll see the Redmine link on the GitLab project pages that takes you to the appropriate Redmine project. + Once you have configured and enabled Redmine you'll see the Redmine link on the GitLab project pages that takes you to the appropriate Redmine project. - As an example, below is a configuration for a project named gitlab-ci. + As an example, below is a configuration for a project named gitlab-ci. -  +  1. To disable the internal issue tracking system in a project, navigate to the General page, expand the [permissions](../settings/index.md#sharing-and-permissions) section and switch the **Issues** toggle to disabled. diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 30940b65454..84adb9637fc 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -58,13 +58,13 @@ Navigate to the webhooks page by going to your project's If you are writing your own endpoint (web server) that will receive GitLab webhooks keep in mind the following things: -- Your endpoint should send its HTTP response as fast as possible. If - you wait too long, GitLab may decide the hook failed and retry it. -- Your endpoint should ALWAYS return a valid HTTP response. If you do - not do this then GitLab will think the hook failed and retry it. - Most HTTP libraries take care of this for you automatically but if - you are writing a low-level hook this is important to remember. -- GitLab ignores the HTTP status code returned by your endpoint. +- Your endpoint should send its HTTP response as fast as possible. If + you wait too long, GitLab may decide the hook failed and retry it. +- Your endpoint should ALWAYS return a valid HTTP response. If you do + not do this then GitLab will think the hook failed and retry it. + Most HTTP libraries take care of this for you automatically but if + you are writing a low-level hook this is important to remember. +- GitLab ignores the HTTP status code returned by your endpoint. ## Secret token diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 3eb5581912a..519c02cf0ad 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -13,7 +13,7 @@ keeping everything in the same place, so that you don't need to jump between different platforms to organize your workflow. With GitLab Issue Boards, you organize your issues in lists that correspond to -their assigned labels, visualizing issues designed as cards throughout that lists. +their assigned labels, visualizing issues designed as cards throughout those lists. You define your process and GitLab organizes it for you. You add your labels then create the corresponding list to pull in your existing issues. When @@ -27,12 +27,9 @@ Issue Boards** (version introduced in GitLab 8.11 - August 2016). ### Advanced features of Issue Boards -With [GitLab Starter](https://about.gitlab.com/pricing/), you can create -[multiple issue boards](#multiple-issue-boards-starter) for a given project. **(STARTER)** - -With [GitLab Premium](https://about.gitlab.com/pricing/), you can also create multiple -issue boards for your groups, and add lists for [assignees](#assignee-lists-premium) and -[milestones](#milestone-lists-premium). **(PREMIUM)** +- Create multiple issue boards per project. +- Create multiple issue boards per group. **(PREMIUM)** +- Add lists for [assignees](#assignee-lists-premium) and [milestones](#milestone-lists-premium). **(PREMIUM)** Check all the [advanced features of Issue Boards](#gitlab-enterprise-features-for-issue-boards) below. @@ -58,8 +55,7 @@ You create issues, host code, perform reviews, build, test, and deploy from one single platform. Issue Boards help you to visualize and manage the entire process _in_ GitLab. -With [Multiple Issue Boards](#use-cases-for-multiple-issue-boards), available -only in [different tiers of GitLab Enterprise Edition](#gitlab-enterprise-features-for-issue-boards), +With [Multiple Issue Boards](#use-cases-for-multiple-issue-boards), you go even further, as you can not only keep yourself and your project organized from a broader perspective with one Issue Board per project, but also allow your team members to organize their own workflow by creating @@ -88,7 +84,7 @@ If we have the labels "**backend**", "**frontend**", "**staging**", and "**production**", and an Issue Board with a list for each, we can: - Visualize the entire flow of implementations since the - beginning of the development lifecycle until deployed to production + beginning of the development life cycle until deployed to production - Prioritize the issues in a list by moving them vertically - Move issues between lists to organize them according to the labels you've set - Add multiple issues to lists in the board by selecting one or more existing issues @@ -97,8 +93,7 @@ If we have the labels "**backend**", "**frontend**", "**staging**", and ### Use cases for Multiple Issue Boards -With [Multiple Issue Boards](#multiple-issue-boards-starter), available only in -[GitLab Enterprise Edition](https://about.gitlab.com/pricing/), +With [Multiple Issue Boards](#multiple-issue-boards), each team can have their own board to organize their workflow individually. #### Scrum team @@ -143,10 +138,10 @@ Create lists for each of your team members and quickly drag-and-drop issues onto - **Issue Board** - Each board represents a unique view for your issues. It can have multiple lists with each list consisting of issues represented by cards. - **List** - A column on the issue board that displays issues matching certain attributes. In addition to the default lists of 'Open' and 'Closed' issue, each additional list will show issues matching your chosen label or assignee. On the top of that list you can see the number of issues that belong to it. - - **Label list**: a list based on a label. It shows all opened issues with that label. - - **Assignee list**: a list which includes all issues assigned to a user. - - **Open** (default): shows all open issues that do not belong to one of the other lists. Always appears as the leftmost list. - - **Closed** (default): shows all closed issues. Always appears as the rightmost list. + - **Label list**: a list based on a label. It shows all opened issues with that label. + - **Assignee list**: a list which includes all issues assigned to a user. + - **Open** (default): shows all open issues that do not belong to one of the other lists. Always appears as the leftmost list. + - **Closed** (default): shows all closed issues. Always appears as the rightmost list. - **Card** - A box in the list that represents an individual issue. The information you can see on a card consists of the issue number, the issue title, the assignee, and the labels associated with the issue. You can drag cards from one list to another to change their label or assignee from that of the source list to that of the destination list. ## Permissions @@ -159,13 +154,14 @@ Issue Board, that is, create or delete lists and drag issues from one list to an GitLab Issue Boards are available on GitLab Core and GitLab.com Free, but some advanced functionalities are only present in higher tiers: GitLab.com Bronze, Silver, or Gold, or GitLab self-managed Starter, Premium, and Ultimate, as described -on the following sections. +in the following sections. For a collection of [features per tier](#summary-of-features-per-tier), check the summary below. -### Multiple Issue Boards **(STARTER)** +### Multiple Issue Boards -> Introduced in [GitLab Enterprise Edition 8.13](https://about.gitlab.com/2016/10/22/gitlab-8-13-released/#multiple-issue-boards-ee). +> - Multiple Issue Boards per project [moved](https://gitlab.com/gitlab-org/gitlab-ce/issues/53811) to [GitLab Core](https://about.gitlab.com/pricing/) in GitLab 12.1. +> - Multiple Issue Boards per group is available in [GitLab Premium Edition](https://about.gitlab.com/pricing/). Multiple Issue Boards, as the name suggests, allow for more than one Issue Board for a given project or group. This is great for large projects with more than one team @@ -184,10 +180,6 @@ These are shortcuts to your last 4 visited boards. When you're revisiting an issue board in a project or group with multiple boards, GitLab will automatically load the last board you visited. -NOTE: **Note:** -The Multiple Issue Boards feature is available for -**projects in GitLab Starter Edition** and for **groups in GitLab Premium Edition**. - ### Configurable Issue Boards **(STARTER)** > Introduced in [GitLab Starter Edition 10.2](https://about.gitlab.com/2017/11/22/gitlab-10-2-released/#issue-boards-configuration). @@ -386,6 +378,10 @@ a given board inside your GitLab instance, any time those two issues are subsequ loaded in any board in the same instance (could be a different project board or a different group board, for example), that ordering will be maintained. +This ordering also affects [issue lists](issues/sorting_issue_lists.md). +Changing the order in an issue board changes the ordering in an issue list, +and vice versa. + ### Filtering issues You should be able to use the filters on top of your Issue Board to show only diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index 2c755e0fb4d..c10ef564f0d 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -77,3 +77,49 @@ project's search results respectively. | Maintainer access | Guest access | | :-----------: | :----------: | |  |  | + +## Merge Requests for Confidential Issues + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/58583) in GitLab 12.1. + +To help prevent confidential information being leaked from a public project +in the process of resolving a confidential issue, confidential issues can be +resolved by creating a merge request from a private fork. + +The merge request created will target the default branch of the private fork, +not the default branch of the public upstream project. This prevents the merge +request, branch, and commits entering the public repository, and revealing +confidential information prematurely. When the confidential commits are ready +to be made public, this can be done by opening a merge request from the private +fork to the public upstream project. + +TIP: **Best practice:** +If you create a long-lived private fork in the same group or in a sub-group of +the original upstream, all the users with Developer membership to the public +project will also have the same permissions in the private project. This way, +all the Developers, who have access to view confidential issues, will have a +streamlined workflow for fixing them. + +### How it works + +On a confidential issue, a **Create confidential merge request** button is +available. Clicking on it will open a dropdown where you can choose to +**Create confidential merge request and branch** or **Create branch**: + +| Create confidential merge request | Create branch | +| :-------------------------------: | :-----------: | +|  |  | + +The **Project** dropdown includes the list of private forks the user is a member +of as at least a Developer and merge requests are enabled. + +Whenever the **Branch name** and **Source (branch or tag)** fields change, the +availability of the target or source branch will be checked. Both branches should +be available in the private fork selected. + +By clicking the **Create confidential merge request** button, GitLab will create +the branch and merge request in the private fork. When you choose +**Create branch**, GitLab will only create the branch. + +Once the branch is created in the private fork, developers can now push code to +that branch to fix the confidential issue. diff --git a/doc/user/project/issues/crosslinking_issues.md b/doc/user/project/issues/crosslinking_issues.md index 93dc2a2e4ca..ec9eb7b62bb 100644 --- a/doc/user/project/issues/crosslinking_issues.md +++ b/doc/user/project/issues/crosslinking_issues.md @@ -47,7 +47,7 @@ display in both issues. The same is valid when mentioning issues in [merge reque ## From Merge Requests Mentioning issues in merge request comments works exactly the same way as -they do for [related issues](#from-related-issues). +they do for [related issues](#from-related-issues). When you mention an issue in a merge request description, it will simply [link the issue and merge request together](#from-related-issues). Additionally, diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md index 0b7a7af5927..27ce3a814f9 100644 --- a/doc/user/project/issues/csv_export.md +++ b/doc/user/project/issues/csv_export.md @@ -48,7 +48,6 @@ Exported issues are always sorted by `Issue ID`. Data will be encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows. The first row will be the headers, which are listed in the following table along with a description of the values: - | Column | Description | |---------|-------------| | Issue ID | Issue `iid` | @@ -71,7 +70,6 @@ Data will be encoded with a comma as the column delimiter, with `"` used to quot | Time Estimate | [Time estimate](../../../workflow/time_tracking.md#estimates) in seconds | | Time Spent | [Time spent](../../../workflow/time_tracking.md#time-spent) in seconds | - ## Limitations -As the issues will be sent as an email attachment, there is a limit on how much data can be exported. Currently this limit is 20MB to ensure successful delivery across a range of email providers. If this limit is reached we suggest narrowing the search before export, perhaps by exporting open and closed issues separately. +As the issues will be sent as an email attachment, there is a limit on how much data can be exported. Currently this limit is 15MB to ensure successful delivery across a range of email providers. If this limit is reached we suggest narrowing the search before export, perhaps by exporting open and closed issues separately. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md new file mode 100644 index 00000000000..1324a90e00b --- /dev/null +++ b/doc/user/project/issues/design_management.md @@ -0,0 +1,79 @@ +# Design Management **(PREMIUM)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. + +CAUTION: **Warning:** +This an **alpha** feature and is subject to change at any time without +prior notice. + +## Overview + +Design Management allows you to upload design assets (wireframes, mockups, etc.) +to GitLab issues and keep them stored in one single place, accessed by the Design +Management's page within an issue, giving product designers, product managers, and engineers a +way to collaborate on designs over one single source of truth. + +You can easily share mock-ups of designs with your team, or visual regressions can be easily +viewed and addressed. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see the video [Design Management (GitLab 12.2)](https://www.youtube.com/watch?v=CCMtCqdK_aM). + +## Requirements + +Design Management requires +[Large File Storage (LFS)](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +to be enabled: + +- For GitLab.com, LFS is already enabled. +- For self-managed instances, a GitLab administrator must have + [enabled LFS globally](../../../workflow/lfs/lfs_administration.md). +- For both GitLab.com and self-managed instances: LFS must be enabled for the project itself. + If enabled globally, LFS will be enabled by default to all projects. To enable LFS on the + project level, navigate to your project's **Settings > General**, expand **Visibility, project features, permissions** + and enable **Git Large File Storage**. + +## Limitations + +- Files uploaded must have a file extension of either `png`, `jpg`, `jpeg`, `gif`, `bmp`, `tiff` or `ico`. + The [`svg` extension is not yet supported](https://gitlab.com/gitlab-org/gitlab-ee/issues/12771). +- Design uploads are limited to 10 files at a time. +- [Designs cannot yet be deleted](https://gitlab.com/gitlab-org/gitlab-ee/issues/11089). +- Design Management is + [not yet supported in the project export](https://gitlab.com/gitlab-org/gitlab-ee/issues/11090). +- Design Management data + [isn't deleted when a project is destroyed](https://gitlab.com/gitlab-org/gitlab-ee/issues/13429) yet. +- Design Management data [won't be moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/13426) + when an issue is moved, nor [deleted](https://gitlab.com/gitlab-org/gitlab-ee/issues/13427) + when an issue is deleted. +- Design Management + [isn't supported by Geo](https://gitlab.com/groups/gitlab-org/-/epics/1633) yet. + +## The Design Management page + +Navigate to the **Design Management** page from any issue by clicking the **Designs** tab: + + + +## Adding designs + +To upload design images, click the **Upload Designs** button and select images to upload. + +Designs with the same filename as an existing uploaded design will create a new version +of the design, and will replace the previous version. + +## Viewing designs + +Images on the Design Management page can be enlarged by clicking on them. + +## Adding annotations to designs + +When a design image is displayed, you can add annotations to it by clicking on +the image. A badge is added to the image and a form is displayed to start a new +discussion. For example: + + + +When submitted, the form saves a badge linked to the discussion on the image. Different discussions have different badge numbers. For example: + + diff --git a/doc/user/project/issues/img/adding_note_to_design_1.png b/doc/user/project/issues/img/adding_note_to_design_1.png Binary files differnew file mode 100644 index 00000000000..aa50bbb69ce --- /dev/null +++ b/doc/user/project/issues/img/adding_note_to_design_1.png diff --git a/doc/user/project/issues/img/adding_note_to_design_2.png b/doc/user/project/issues/img/adding_note_to_design_2.png Binary files differnew file mode 100644 index 00000000000..37cefeb1a15 --- /dev/null +++ b/doc/user/project/issues/img/adding_note_to_design_2.png diff --git a/doc/user/project/issues/img/comment-or-discussion.png b/doc/user/project/issues/img/comment-or-discussion.png Binary files differindex ccecc9fa39f..a29014c984c 100644 --- a/doc/user/project/issues/img/comment-or-discussion.png +++ b/doc/user/project/issues/img/comment-or-discussion.png diff --git a/doc/user/project/issues/img/confidential_mr_branch_dropdown_v12_1.png b/doc/user/project/issues/img/confidential_mr_branch_dropdown_v12_1.png Binary files differnew file mode 100644 index 00000000000..1f4ad5c42bb --- /dev/null +++ b/doc/user/project/issues/img/confidential_mr_branch_dropdown_v12_1.png diff --git a/doc/user/project/issues/img/confidential_mr_dropdown_v12_1.png b/doc/user/project/issues/img/confidential_mr_dropdown_v12_1.png Binary files differnew file mode 100644 index 00000000000..7b7bd599a71 --- /dev/null +++ b/doc/user/project/issues/img/confidential_mr_dropdown_v12_1.png diff --git a/doc/user/project/issues/img/create_mr_from_issue.png b/doc/user/project/issues/img/create_mr_from_issue.png Binary files differindex 680c42b5df0..d05a678cd17 100644 --- a/doc/user/project/issues/img/create_mr_from_issue.png +++ b/doc/user/project/issues/img/create_mr_from_issue.png diff --git a/doc/user/project/issues/img/design_management_v12_2.png b/doc/user/project/issues/img/design_management_v12_2.png Binary files differnew file mode 100644 index 00000000000..ad803df4e63 --- /dev/null +++ b/doc/user/project/issues/img/design_management_v12_2.png diff --git a/doc/user/project/issues/img/issues_main_view_numbered.png b/doc/user/project/issues/img/issues_main_view_numbered.png Binary files differindex 16cb6b497b0..92b9df44972 100644 --- a/doc/user/project/issues/img/issues_main_view_numbered.png +++ b/doc/user/project/issues/img/issues_main_view_numbered.png diff --git a/doc/user/project/issues/img/reopen-issue.png b/doc/user/project/issues/img/reopen-issue.png Binary files differindex 1749be31239..fc48742afe0 100644 --- a/doc/user/project/issues/img/reopen-issue.png +++ b/doc/user/project/issues/img/reopen-issue.png diff --git a/doc/user/project/issues/img/report-abuse.png b/doc/user/project/issues/img/report-abuse.png Binary files differindex f189cbf1d36..f8cef22da03 100644 --- a/doc/user/project/issues/img/report-abuse.png +++ b/doc/user/project/issues/img/report-abuse.png diff --git a/doc/user/project/issues/img/show-all-activity.png b/doc/user/project/issues/img/show-all-activity.png Binary files differindex c43ba75ce25..55c6f5ab5db 100644 --- a/doc/user/project/issues/img/show-all-activity.png +++ b/doc/user/project/issues/img/show-all-activity.png diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index e917697e973..bf04ed2d2d0 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -89,7 +89,7 @@ Key actions for Issues include:  On an issue's page, you can view [all aspects of the issue](issue_data_and_actions.md), -and modify them if you you have the necessary [permissions](../../permissions.md). +and modify them if you have the necessary [permissions](../../permissions.md). ### Issues list @@ -104,7 +104,8 @@ view, you can also make certain changes [in bulk](../bulk_editing.md) to the dis For more information, see the [Issue Data and Actions](issue_data_and_actions.md) page for a rundown of all the fields and information in an issue. -For sorting by issue priority, see [Label Priority](../labels.md#label-priority). +You can sort a list of issues several ways, including by issue creation date, milestone due date, +etc. For more information, see the [Sorting and Ordering Issue Lists](sorting_issue_lists.md) page. ### Issue boards @@ -119,6 +120,12 @@ associated label or assignee will change to match that of the new column. The en board can also be filtered to only include issues from a certain milestone or an overarching label. +### Design Management **(PREMIUM)** + +With [Design Management](design_management.md), you can upload design +assets to issues and view them all together to easily share and +collaborate with your team. + ### Epics **(ULTIMATE)** [Epics](../../group/epics/index.md) let you manage your portfolio of projects more diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index 9c4d0de46d3..d7d168710ef 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -50,7 +50,12 @@ The button to do this has a different label depending on whether the issue is al #### 3. Assignee -An issue can be assigned to yourself, another person, or [many people](#31-multiple-assignees-STARTER). +An issue can be assigned to: + +- Yourself. +- Another person. +- [Many people](#31-multiple-assignees-STARTER). **(STARTER)** + The assignee(s) can be changed as often as needed. The idea is that the assignees are responsible for that issue until it's reassigned to someone else to take it from there. When assigned to someone, it will appear in their assigned issues list. @@ -116,13 +121,13 @@ issue boards or the issue list. #### 11. Lock issue -You can [lock the discussions](../../discussions/index.md#lock-discussions) in the issue, +You can [lock the threads](../../discussions/index.md#lock-discussions) in the issue, to prevent further comments from being added. #### 12. Participants -All the users involved in that issue. Either they participated in the [discussion](../../discussions/index.md), -or were mentioned in the description or discussions. +All the users involved in that issue. Either they participated in the [thread](../../discussions/index.md), +or were mentioned in the description or threads. #### 13. Notifications @@ -187,8 +192,8 @@ You can also click the `+` to add more related issues. #### 19. Related Merge Requests -Merge requests that were mentioned in that issue's description or in the issue discussion -thread are listed as [related merge requests](crosslinking_issues.md#from-merge-requests) here. +Merge requests that were mentioned in that issue's description or in the issue thread +are listed as [related merge requests](crosslinking_issues.md#from-merge-requests) here. Also, if the current issue was mentioned as related in another merge request, that merge request will be listed here. @@ -200,17 +205,17 @@ dropdown list of available [GitLab Flavored Markdown Emoji](../../markdown.md#em TIP: **Tip:** Posting "+1" as a comment in a thread spams all subscribed participants of that issue, -clutters the discussion threads, and is not recommended. Awarding an emoji is a way +clutters the threads, and is not recommended. Awarding an emoji is a way to let them know your reaction without spamming them. #### 21. Show all activity You can filter what is displayed in the issue history by clicking on **Show all activity** -and selecting either **Show comments only**, which only shows discussions and hides -updates to the issue, or **Show history only**, which hides discussions and only shows updates. +and selecting either **Show comments only**, which only shows threads and hides +updates to the issue, or **Show history only**, which hides threads and only shows updates. - You can mention a user or a group present in your GitLab instance with - `@username` or `@groupname` and they will be notified via To-Do items + `@username` or `@groupname` and they will be notified via To-Do items and email, unless they have [disabled all notifications](#13-notifications) in their profile settings. - Mentions for yourself (the current logged in user), will be highlighted @@ -242,15 +247,15 @@ filtered, as shown above. Collaborate in the issue by posting comments in its thread. This text field also fully supports [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). -#### 25. Submit Comment, start a discussion, or comment and close +#### 25. Submit Comment, start a thread, or comment and close Once you write a comment, you can: - Click **Comment** and your comment will be published. -- Choose **Start discussion** from the dropdown list and start a new [discussion thread](../../discussions/index.md#threaded-discussions) +- Choose **Start thread** from the dropdown list and start a new [thread](../../discussions/index.md#threaded-discussions) within that issue's main thread to discuss specific points. This invites other participants - to reply directly to your discussion, keeping related comments grouped together. + to reply directly to your thread, keeping related comments grouped together. - + You can also close the issue from here, so you don't need to scroll to the top of the issue page. diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 709588959c1..e9b4e591384 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -7,7 +7,9 @@ you can do with issues. ## Create a new Issue -When you create a new issue, you'll be prompted to fill in the [data and fields of the issue](issue_data_and_actions.md#parts-of-an-issue), as illustrated below. +When you create a new issue, you'll be prompted to fill in the [data and fields of the issue](issue_data_and_actions.md#parts-of-an-issue), as illustrated below. If you know +the values you want to assign to an issue, you can use the [Quick actions](../quick_actions.md) +feature to input values, instead of selecting them from lists.  diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md index 9c72fe33d0d..d7178506b64 100644 --- a/doc/user/project/issues/related_issues.md +++ b/doc/user/project/issues/related_issues.md @@ -19,7 +19,7 @@ Issues from a different project require additional information like the group and the project name. For example: - same project: `#44` -- same group: `project#44 ` +- same group: `project#44` - different group: `group/project#44` Valid references will be added to a temporary list that you can review. diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md new file mode 100644 index 00000000000..0fe86e6f410 --- /dev/null +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -0,0 +1,30 @@ +# Sorting and Ordering Issue Lists + +You can sort a list of issues several ways, including by issue creation date, milestone due date, +etc. The available sorting options can change based on the context of the list. +For sorting by issue priority, see [Label Priority](../labels.md#label-priority). + +In group and project issue lists, it is also possible to order issues manually, +similar to [issue boards](../issue_board.md#issue-ordering-in-a-list). + +## Manual sorting + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/62178) in GitLab 12.2. + +When you select **Manual** sorting, you can change +the order by dragging and dropping the issues. The changed order will persist. Everyone who visits the same list will see the reordered list, with some exceptions. + +Each issue is assigned a relative order value, representing its relative +order with respect to the other issues in the list. When you drag-and-drop reorder +an issue, its relative order value changes accordingly. + +In addition, any time that issue appears in a manually sorted list, +the updated relative order value will be used for the ordering. This means that +if issue `A` is drag-and-drop reordered to be above issue `B` by any user in +a given list inside your GitLab instance, any time those two issues are subsequently +loaded in any list in the same instance (could be a different project issue list or a +different group issue list, for example), that ordering will be maintained. + +This ordering also affects [issue boards](../issue_board.md#issue-ordering-in-a-list). +Changing the order in an issue list changes the ordering in an issue board, +and vice versa. diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index fdfeb4aa4cd..cc1274faa2c 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -2,43 +2,52 @@ ## Overview -Labels allow you to categorize issues or merge requests using descriptive titles like `bug`, `feature request`, or `docs`. Each label also has a customizable color. They allow you to quickly and dynamically filter and manage issues or merge requests you care about, and are visible throughout GitLab in most places where issues and merge requests are located. +Labels allow you to categorize issues or merge requests using descriptive titles like +`bug`, `feature request`, or `docs`. Each label also has a customizable color. They +allow you to quickly and dynamically filter and manage issues or merge requests you +care about, and are visible throughout GitLab in most places where issues and merge +requests are located. ## Project labels and group labels In GitLab, you can create project and group labels: - **Project labels** can be assigned to issues or merge requests in that project only. -- **Group labels** can be assigned to any issue or merge request of any project in that group or any subgroups of the group. +- **Group labels** can be assigned to any issue or merge request in any project in + that group, or any subgroups of the group. ## Scoped labels **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9175) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. -Scoped labels allow teams to use the simple and familiar feature of labels to +Scoped labels allow teams to use the simple and familiar label feature to annotate their issues, merge requests, and epics to achieve custom fields and custom workflow states by leveraging a special label title syntax. -A scoped label is a kind of label defined only by a special double-colon syntax +A scoped label is a kind of label defined by a special double-colon syntax in the label’s title, using the format `key::value`. For example: - + An issue, epic, or merge request cannot have two scoped labels with the same key. -For example, if an issue is already labeled `priority::3` and you apply the label `priority::2` to it, -`priority::3` is automatically removed. +For example, if an issue is already labeled `priority::3`, and then you apply the +`priority::2` label to it, `priority::3` is automatically removed. This functionality is demonstrated in a video titled [Use scoped labels in GitLab 11.10 for custom fields and custom workflows](https://www.youtube.com/watch?v=4BCBby6du3c). ### Labels with multiple colon pairs -If labels have multiple instances of `::`, the longest path from left to right, until the last `::`, is considered the "key" or the "scope". +If labels have multiple instances of `::`, the longest path from left to right, until +the last `::`, is considered the "key" or the "scope". -For example, `nested::key1::value1` and `nested::key1::value2` cannot both exist on the same issue. Adding the latter label will automatically remove the former due to the shared scope of `nested::key1`. +For example, `nested::key1::value1` and `nested::key1::value2` cannot both exist on +the same issue. Adding the latter label will automatically remove the former due to +the shared scope of `nested::key1`. -`nested::key1::value1` and `nested::key2::value1` can both exist on the same issue, as these are considered to use two different label scopes, `nested::key1` and `nested::key2`. +`nested::key1::value1` and `nested::key2::value1` can both exist on the same issue, +as these are considered to use two different label scopes, `nested::key1` and `nested::key2`. -### Workflows with scoped labels **(PREMIUM)** +### Workflows with scoped labels Suppose you wanted a custom field in issues to track the platform operating system that your features target, where each issue should only target one platform. You @@ -58,101 +67,156 @@ be able to advance workflow states consistently in issues themselves. ## Creating labels ->**Note:** +NOTE: **Note:** A permission level of Reporter or higher is required to create labels. ### New project label To create a **project label**, navigate to **Issues > Labels** in the project. -This page only shows project labels in this project and group labels of this project's parent group. +This page only shows the project labels in this project, and the group labels of the +project's parent group. -Click the **New label** button. Enter the title, an optional description, and the background color. Click **Create label** to create the label. +Click the **New label** button. Enter the title, an optional description, and the +background color. Click **Create label** to create the label. -If a project has no labels, you can generate a default set of project labels from its empty label list page: +If a project has no labels, you can generate a default set of project labels from +its empty label list page: - + GitLab will add the following default labels to the project: - + ### New group label -To create a **group label**, follow similar steps from above to project labels. Navigate to **Issues > Labels** in the group and create it from there. -This page only shows group labels in this group. -Alternatively, you can create group labels also from Epic sidebar. Please note that the created label will belong to the immediate group to which epic belongs. +To create a **group label**, navigate to **Issues > Labels** in the **group** and create +it from there. This page only shows group labels in this group. + +Alternatively, you can create group labels from the Epic sidebar. **(ULTIMATE)** - +Please note that the created label will belong to the immediate group to which the +epic belongs. + + Group labels appear in every label list page of the group's child projects. - + ### New project label from sidebar -From the sidebar of an issue or a merge request, you can create a new **project label** inline immediately, instead of navigating to the project label list page. +From the sidebar of an issue or a merge request, you can create a new **project label** +inline immediately, instead of navigating to the project label list page. - + ## Editing labels NOTE: **Note:** A permission level of Reporter or higher is required to edit labels. -You can update a label by navigating to **Issues > Labels** in the project or group and clicking the pencil icon. +To update a label, navigate to **Issues > Labels** in the project or group +and click the pencil icon. The title, description and color can be changed. + +To delete a label, click the three dots next to the `Subscribe` button, and select +**Delete**. -You can delete a label by clicking the trash icon. + ### Promoting project labels to group labels -If you are expanding from a few projects to a larger number of projects within the same group, you may want to share the same label among multiple projects in the same group. If you previously created a project label and now want to make it available for other projects, you can promote it to a group label. +If you are expanding from a few projects to a larger number of projects within the +same group, you may want to share the same label among multiple projects in the same +group. If you previously created a project label and now want to make it available +for other projects, you can promote it to a group label. -From the project label list page, you can promote a project label to a group label. This will merge all project labels across all projects in this group with the same name into a single group label. All issues and merge requests that previously were assigned one of these project labels will now be assigned the new group label. This action cannot be reversed and the changes are permanent. +From the project label list page, you can promote a project label to a group label. +This will merge all project labels with the same name into a single group label, across +all projects in this group. All issues and merge requests that were previously +assigned one of these project labels will now be assigned the new group label. This +action cannot be reversed and the changes are permanent. - + ## Assigning labels from the sidebar -Every issue and merge request can be assigned any number of labels. The labels are visible on every issue and merge request page, in the sidebar. They are also visible in the issue board. From the sidebar, you can assign or unassign a label to the object (i.e. label or unlabel it). You can also perform this as a [quick action](quick_actions.md) in a comment. +Every issue and merge request can be assigned any number of labels. The labels are +visible on every issue and merge request page, in the sidebar. They are also visible on: + +- Every issue and merge request page in the sidebar. +- The issue board. + +From the sidebar, you can assign or unassign a label to the object (i.e. label or +unlabel it). You can also perform this as a [quick action](quick_actions.md), +in a comment. | View labels in sidebar | Assign labels from sidebar | -|:---:|:---:| +|:----------------------:|:--------------------------:| |  |  | ## Searching for project labels -You can search for project labels by navigating from the left sidebar to your -project's **Issues > Labels** and entering your query to the search bar on the -top-right: +To search for project labels, go to **Issues > Labels** in the left sidebar, and enter +your search query in the **Filter** field.  -GitLab will consider the label title and description for the search. +GitLab will check both the label titles and descriptions for the search. + +## Filtering by label -## Filtering issues, merge requests and epics by label +The following can be filtered labels: + +- Issue lists +- Merge Request lists +- Epic lists **(ULTIMATE)** +- Issue Boards ### Filtering in list pages -From the project issue list page and the project merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group (including subgroup ancestors) labels and project labels. +- From the project issue list page and the project merge request list page, you can + [filter](../search/index.md#issues-and-merge-requests) by: + - Group labels (including subgroup ancestors) + - Project labels -From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels (including subgroup ancestors and subgroup descendants) and project labels. +- From the group issue list page and the group merge request list page, you can + [filter](../search/index.md#issues-and-merge-requests) by: + - Group labels (including subgroup ancestors and subgroup descendants) + - Project labels -From the group epic list page, you can [filter](../search/index.md#issues-and-merge-requests) by both current group labels as well as descendant group labels. +- You can [filter](../search/index.md#issues-and-merge-requests) the group epic list + page by: **(ULTIMATE)** + - Current group labels + - Descendant group labels - + ### Filtering in issue boards -- From [project boards](issue_board.md), you can filter by both group labels and project labels in the [search and filter bar](../search/index.md#issue-boards). -- From [group issue boards](issue_board.md#group-issue-boards-premium), you can filter by only group labels in the [search and filter bar](../search/index.md#issue-boards). **(PREMIUM)** -- From [project boards](issue_board.md), you can filter by both group labels and project labels in the [issue board configuration](issue_board.md#configurable-issue-boards-starter). **(STARTER)** -- From [group issue boards](issue_board.md#group-issue-boards-premium), you can filter by only group labels in the [issue board configuration](issue_board.md#configurable-issue-boards-starter). **(STARTER)** +- From [project boards](issue_board.md), you can use the [search and filter bar](../search/index.md#issue-boards) + to filter by: + - Group labels + - Project labels + +- From [group issue boards](issue_board.md#group-issue-boards-premium), you can use the + [search and filter bar](../search/index.md#issue-boards) to filter by group labels only. **(PREMIUM)** + +- From [project boards](issue_board.md), in the [issue board configuration](issue_board.md#configurable-issue-boards-starter), + you can filter by: **(STARTER)** + - Group labels + - Project labels + +- From [group issue boards](issue_board.md#group-issue-boards-premium), in the [issue board configuration](issue_board.md#configurable-issue-boards-starter), + you can filter by group labels only. **(STARTER)** ## Subscribing to labels -From the project label list page and the group label list page, you can subscribe to [notifications](../../workflow/notifications.md) of a given label, to alert you that the label has been assigned to an issue or merge request. +From the project label list page and the group label list page, you can subscribe +to [notifications](../../workflow/notifications.md) of a given label, to alert you +that the label has been assigned to an issue or merge request. - + ## Label priority @@ -161,26 +225,43 @@ From the project label list page and the group label list page, you can subscrib > - Introduced in GitLab 8.9. > - Priority sorting is based on the highest priority label only. [This discussion](https://gitlab.com/gitlab-org/gitlab-ce/issues/18554) considers changing this. -Labels can have relative priorities, which are used in the "Label priority" and "Priority" sort orders of the issue and merge request list pages. +Labels can have relative priorities, which are used in the "Label priority" and +"Priority" sort orders of the issue and merge request list pages. + +From the project label list page, star a label to indicate that it has a priority. + + + +Drag starred labels up and down the list to change their priority. Higher in the list +means higher priority. Prioritization happens at the project level, only on the project +label list page, and not on the group label list page. -From the project label list page, star a label to indicate that it has a priority. Drag starred labels up and down to change their priority. Higher means higher priority. Prioritization happens at the project level, only on the project label list page, and not on the group label list page. However, both project and group labels can be prioritized on the project label list page since both types are displayed on the project label list page. +However, both project and group +labels can be prioritized on the project label list page since both types are displayed +on the project label list page. - + -On the project and group issue and merge request list pages, you can sort by `Label priority` and `Priority`, which account for objects (issues and merge requests) that have prioritized labels assigned to them. +On the merge request and issue pages, for both groups and projects, you can sort by `Label priority` +and `Priority`, which account for objects (issues and merge requests) that have prioritized +labels assigned to them. If you sort by `Label priority`, GitLab considers this sort comparison order: - Object with a higher priority prioritized label. - Object without a prioritized label. -Ties are broken arbitrarily. (Note that we _only_ consider the highest prioritized label in an object, and not any of the lower prioritized labels. [This discussion](https://gitlab.com/gitlab-org/gitlab-ce/issues/18554) considers changing this.) +Ties are broken arbitrarily. Note that we _only_ consider the highest prioritized label +in an object, and not any of the lower prioritized labels. [This discussion](https://gitlab.com/gitlab-org/gitlab-ce/issues/18554) +considers changing this.  If you sort by `Priority`, GitLab considers this sort comparison order: -- Object's assigned [milestone](milestones/index.md)'s due date is sooner, provided the object has a milestone and the milestone has a due date. If this isn't the case, consider the object having a due date in the infinite future. +- Due date of the assigned [milestone](milestones/index.md) is sooner, provided + the object has a milestone and the milestone has a due date. If this isn't the case, + consider the object having a due date in the infinite future. - Object with a higher priority prioritized label. - Object without a prioritized label. diff --git a/doc/user/project/members/img/access_requests_management.png b/doc/user/project/members/img/access_requests_management.png Binary files differindex 8996d9564d7..9a1c9621e41 100644 --- a/doc/user/project/members/img/access_requests_management.png +++ b/doc/user/project/members/img/access_requests_management.png diff --git a/doc/user/project/members/img/request_access_button.png b/doc/user/project/members/img/request_access_button.png Binary files differindex e8b490b91b8..e693f9a9ac2 100644 --- a/doc/user/project/members/img/request_access_button.png +++ b/doc/user/project/members/img/request_access_button.png diff --git a/doc/user/project/members/img/withdraw_access_request_button.png b/doc/user/project/members/img/withdraw_access_request_button.png Binary files differindex 6a3172dfcdb..e5a8fe0b356 100644 --- a/doc/user/project/members/img/withdraw_access_request_button.png +++ b/doc/user/project/members/img/withdraw_access_request_button.png diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 2a490e3fd7f..e343fd45488 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -10,8 +10,6 @@ or import a new user to your project. To view, edit, add, and remove project's members, go to your project's **Settings > Members**. ---- - ## Add a user Right next to **People**, start typing the name or username of the user you @@ -19,22 +17,16 @@ want to add.  ---- - Select the user and the [permission level](../../permissions.md) that you'd like to give the user. Note that you can select more than one user.  ---- - Once done, hit **Add users to project** and they will be immediately added to your project with the permissions you gave them above.  ---- - From there on, you can either remove an existing user or change their access level to the project. @@ -47,8 +39,6 @@ In the dropdown menu, you can see only the projects you are Maintainer on.  ---- - Select the one you want and hit **Import project members**. A flash message notifying you that the import was successful will appear, and the new members are now in the project's members list. Notice that the permissions that they @@ -56,8 +46,6 @@ had on the project you imported from are retained.  ---- - ## Invite people using their e-mail address If a user you want to give access to doesn't have an account on your GitLab @@ -66,23 +54,17 @@ user search field.  ---- - As you can imagine, you can mix inviting multiple people and adding existing GitLab users to the project.  ---- - Once done, hit **Add users to project** and watch that there is a new member with the e-mail address we used above. From there on, you can resend the invitation, change their access level, or even delete them.  ---- - Once the user accepts the invitation, they will be prompted to create a new GitLab account using the same e-mail address the invitation was sent to. @@ -97,15 +79,11 @@ side of your screen.  ---- - Project owners & maintainers will be notified of your request and will be able to approve or decline it on the members page.  ---- - If you change your mind before your request is approved, just click the **Withdraw Access Request** button. diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 611ff0e6bfb..9340d239677 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -20,18 +20,18 @@ To share 'Project Acme' with the 'Engineering' group: 1. For 'Project Acme' use the left navigation menu to go to **Settings > Members** -  +  1. Select the 'Share with group' tab 1. Add the 'Engineering' group with the maximum access level of your choice 1. Click **Share** to share it -  +  1. After sharing 'Project Acme' with 'Engineering', the project will be listed on the group dashboard -  +  Note that you can only share a project with: diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index ad1d79ae5b1..3a409bab19d 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -1,5 +1,6 @@ --- type: reference, howto +disqus_identifier: 'https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html' --- # Code Quality **(STARTER)** @@ -9,8 +10,18 @@ in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. With the help of [GitLab CI/CD](../../../ci/README.md), you can analyze your source code quality using GitLab Code Quality. -Code Quality uses [Code Climate Engines](https://codeclimate.com), which are -free and open source. Code Quality doesn't require a Code Climate subscription. + +Code Quality: + +- Uses [Code Climate Engines](https://codeclimate.com), which are + free and open source. Code Quality doesn't require a Code Climate + subscription. +- Runs in [pipelines](../../../ci/pipelines.md) using an Docker image built in + [GitLab Code + Quality](https://gitlab.com/gitlab-org/security-products/codequality) project. +- Can make use of a [template](#template-and-examples). +- Is available with [Auto + DevOps](../../../topics/autodevops/index.md#auto-code-quality-starter). Going a step further, GitLab can show the Code Quality report right in the merge request widget area: @@ -21,22 +32,48 @@ in the merge request widget area: For instance, consider the following workflow: -1. Your backend team member starts a new implementation for making a certain feature in your app faster -1. With Code Quality reports, they analyze how their implementation is impacting the code quality -1. The metrics show that their code degrade the quality in 10 points -1. You ask a co-worker to help them with this modification -1. They both work on the changes until Code Quality report displays no degradations, only improvements -1. You approve the merge request and authorize its deployment to staging -1. Once verified, their changes are deployed to production +1. Your backend team member starts a new implementation for making a certain + feature in your app faster. +1. With Code Quality reports, they analyze how their implementation is impacting + the code quality. +1. The metrics show that their code degrade the quality in 10 points. +1. You ask a co-worker to help them with this modification. +1. They both work on the changes until Code Quality report displays no + degradations, only improvements. +1. You approve the merge request and authorize its deployment to staging. +1. Once verified, their changes are deployed to production. + +## Template and examples + +For most GitLab instances, the supplied template is the preferred method of +implementing Code Quality. See +[Analyze your project's Code Quality](../../../ci/examples/code_quality.md) for: + +- Information on the builtin GitLab Code Quality template. +- Examples of manual GitLab configuration for earlier GitLab versions. -## How it works +## Configuring jobs using variables -First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the -[Code Quality report artifact](../../../ci/yaml/README.md#artifactsreportscodequality-starter). +The Code Quality job supports environment variables that users can set to +configure job execution at runtime. -The Code Quality report artifact is a subset of the -[Code Climate spec](https://github.com/codeclimate/spec/blob/master/SPEC.md#data-types). -It must be a JSON file containing an array of objects with the following properties: +For a list of available environment variables, see +[Environment variables](https://gitlab.com/gitlab-org/security-products/codequality/blob/master/README.md#environment-variables). + +## Implementing a custom tool + +It's possible to have a custom tool provide Code Quality reports in GitLab. To +do this: + +1. Define a job in your `.gitlab-ci.yml` file that generates the + [Code Quality report + artifact](../../../ci/yaml/README.md#artifactsreportscodequality-starter). +1. Configure your tool to generate the Code Quality report artifact as a JSON + file that implements subset of the [Code Climate + spec](https://github.com/codeclimate/spec/blob/master/SPEC.md#data-types). + +The Code Quality report artifact JSON file must contain an array of objects +with the following properties: | Name | Description | | ---------------------- | -------------------------------------------------------------------------------------- | @@ -63,13 +100,16 @@ Example: ``` NOTE: **Note:** -Although the Code Climate spec supports more properties, those are ignored by GitLab. +Although the Code Climate spec supports more properties, those are ignored by +GitLab. + +## Code Quality reports -For more information on what the Code Quality job should look like, check the -example on [analyzing a project's code quality](../../../ci/examples/code_quality.md). +Once the Code Quality job has completed, GitLab: -GitLab then checks this report, compares the metrics between the source and target -branches, and shows the information right on the merge request. +- Checks the generated report. +- Compares the metrics between the source and target branches. +- Shows the information right on the merge request. If multiple jobs in a pipeline generate a code quality artifact, only the artifact from the last created job (the job with the largest job ID) is used. To avoid confusion, diff --git a/doc/user/project/merge_requests/code_quality_diff.md b/doc/user/project/merge_requests/code_quality_diff.md index ccc694672a6..0aa108a2e73 100644 --- a/doc/user/project/merge_requests/code_quality_diff.md +++ b/doc/user/project/merge_requests/code_quality_diff.md @@ -1,5 +1,4 @@ --- -redirect_from: 'code_quality_diff.md' redirect_to: 'code_quality.md' --- diff --git a/doc/user/project/merge_requests/fast_forward_merge.md b/doc/user/project/merge_requests/fast_forward_merge.md index 4f3c68090e4..2d59e535ce1 100644 --- a/doc/user/project/merge_requests/fast_forward_merge.md +++ b/doc/user/project/merge_requests/fast_forward_merge.md @@ -15,7 +15,7 @@ to accept merge requests without creating merge commits. When the fast-forward merge ([`--ff-only`](https://git-scm.com/docs/git-merge#git-merge---ff-only)) setting is enabled, no merge commits will be created and all merges are fast-forwarded, -which means that merging is only allowed if the branch could be fast-forwarded. +which means that merging is only allowed if the branch can be fast-forwarded. When a fast-forward merge is not possible, the user is given the option to rebase. @@ -28,9 +28,15 @@ When a fast-forward merge is not possible, the user is given the option to rebas Now, when you visit the merge request page, you will be able to accept it **only if a fast-forward merge is possible**. + + +If a fast-forward merge is not possible but a conflict free rebase is possible, +a rebase button will be offered. +  -If the target branch is ahead of the source branch, you need to rebase the +If the target branch is ahead of the source branch and a conflict free rebase is +not possible, you need to rebase the source branch locally before you will be able to do a fast-forward merge.  diff --git a/doc/user/project/merge_requests/img/allow_collaboration.png b/doc/user/project/merge_requests/img/allow_collaboration.png Binary files differindex e40e8a6b11c..cc13493646d 100644 --- a/doc/user/project/merge_requests/img/allow_collaboration.png +++ b/doc/user/project/merge_requests/img/allow_collaboration.png diff --git a/doc/user/project/merge_requests/img/allow_collaboration_after_save.png b/doc/user/project/merge_requests/img/allow_collaboration_after_save.png Binary files differindex 4ba4c84c8c5..bc7678b21ec 100644 --- a/doc/user/project/merge_requests/img/allow_collaboration_after_save.png +++ b/doc/user/project/merge_requests/img/allow_collaboration_after_save.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png Binary files differindex b6dc86f312e..2598cc71c33 100644 --- a/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png +++ b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png Binary files differdeleted file mode 100644 index b6f6188b9cd..00000000000 --- a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png Binary files differnew file mode 100644 index 00000000000..32b9a3b9ce4 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_project_edit_v12_3.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_edit.png b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png Binary files differindex 868b9d58740..4c554d846cc 100644 --- a/doc/user/project/merge_requests/img/approvals_starter_project_edit.png +++ b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_empty.png b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png Binary files differindex 7375820224c..fc88a59a745 100644 --- a/doc/user/project/merge_requests/img/approvals_starter_project_empty.png +++ b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png diff --git a/doc/user/project/merge_requests/img/approve.png b/doc/user/project/merge_requests/img/approve.png Binary files differindex e68259ac5c2..e2641f48c7a 100644 --- a/doc/user/project/merge_requests/img/approve.png +++ b/doc/user/project/merge_requests/img/approve.png diff --git a/doc/user/project/merge_requests/img/approve_additionally.png b/doc/user/project/merge_requests/img/approve_additionally.png Binary files differindex 3db5a9159e5..bab0cd4e041 100644 --- a/doc/user/project/merge_requests/img/approve_additionally.png +++ b/doc/user/project/merge_requests/img/approve_additionally.png diff --git a/doc/user/project/merge_requests/img/create_from_email.png b/doc/user/project/merge_requests/img/create_from_email.png Binary files differindex 5cb2afaf976..14eef473e27 100644 --- a/doc/user/project/merge_requests/img/create_from_email.png +++ b/doc/user/project/merge_requests/img/create_from_email.png diff --git a/doc/user/project/merge_requests/img/cross_project_dependencies_edit_inaccessible_v12_2.png b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_inaccessible_v12_2.png Binary files differnew file mode 100644 index 00000000000..2dc02634fd8 --- /dev/null +++ b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_inaccessible_v12_2.png diff --git a/doc/user/project/merge_requests/img/cross_project_dependencies_edit_v12_2.png b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_v12_2.png Binary files differnew file mode 100644 index 00000000000..362e7e0ead2 --- /dev/null +++ b/doc/user/project/merge_requests/img/cross_project_dependencies_edit_v12_2.png diff --git a/doc/user/project/merge_requests/img/cross_project_dependencies_view_v12_2.png b/doc/user/project/merge_requests/img/cross_project_dependencies_view_v12_2.png Binary files differnew file mode 100644 index 00000000000..e00231c839b --- /dev/null +++ b/doc/user/project/merge_requests/img/cross_project_dependencies_view_v12_2.png diff --git a/doc/user/project/merge_requests/img/ff_merge_mr.png b/doc/user/project/merge_requests/img/ff_merge_mr.png Binary files differnew file mode 100644 index 00000000000..241cc990343 --- /dev/null +++ b/doc/user/project/merge_requests/img/ff_merge_mr.png diff --git a/doc/user/project/merge_requests/img/filter_approver_merge_requests.png b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png Binary files differindex 9c386391a4f..4c28ee17f47 100644 --- a/doc/user/project/merge_requests/img/filter_approver_merge_requests.png +++ b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png diff --git a/doc/user/project/merge_requests/img/filter_wip_merge_requests.png b/doc/user/project/merge_requests/img/filter_wip_merge_requests.png Binary files differindex 8df6a3c9a29..0989b41e2a4 100644 --- a/doc/user/project/merge_requests/img/filter_wip_merge_requests.png +++ b/doc/user/project/merge_requests/img/filter_wip_merge_requests.png diff --git a/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png b/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png Binary files differnew file mode 100644 index 00000000000..ee94dbdea5c --- /dev/null +++ b/doc/user/project/merge_requests/img/incrementally_expand_merge_request_diffs_v12_2.png diff --git a/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png b/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png Binary files differindex ea3aff59aa1..ed374b11fbd 100644 --- a/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png +++ b/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png diff --git a/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png b/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png Binary files differindex 9ae6e350798..dde2680ed74 100644 --- a/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png +++ b/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png diff --git a/doc/user/project/merge_requests/img/remove_approval.png b/doc/user/project/merge_requests/img/remove_approval.png Binary files differindex 6083e1745ef..b178d26cf85 100644 --- a/doc/user/project/merge_requests/img/remove_approval.png +++ b/doc/user/project/merge_requests/img/remove_approval.png diff --git a/doc/user/project/merge_requests/img/squash_mr_message.png b/doc/user/project/merge_requests/img/squash_mr_message.png Binary files differindex 8734cab29aa..8c7dc7886f7 100644 --- a/doc/user/project/merge_requests/img/squash_mr_message.png +++ b/doc/user/project/merge_requests/img/squash_mr_message.png diff --git a/doc/user/project/merge_requests/img/wip_blocked_accept_button.png b/doc/user/project/merge_requests/img/wip_blocked_accept_button.png Binary files differindex b6d38d85165..ab2c8425b83 100644 --- a/doc/user/project/merge_requests/img/wip_blocked_accept_button.png +++ b/doc/user/project/merge_requests/img/wip_blocked_accept_button.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 37a0630d0f3..04db54872d3 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -18,7 +18,7 @@ It is as simple as the name implies: a _request_ to _merge_ one branch into anot With GitLab merge requests, you can: - Compare the changes between two [branches](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell#_git_branching) -- [Review and discuss](../../discussions/index.md#discussions) the proposed modifications inline +- [Review and discuss](../../discussions/index.md#threads) the proposed modifications inline - Live preview the changes when [Review Apps](../../../ci/review_apps/index.md) is configured for your project - Build, test, and deploy your code in a per-branch basis with built-in [GitLab CI/CD](../../../ci/README.md) - Prevent the merge request from being merged before it's ready with [WIP MRs](#work-in-progress-merge-requests) @@ -41,12 +41,13 @@ With **[GitLab Enterprise Edition][ee]**, you can also: - View the deployment process across projects with [Multi-Project Pipelines](../../../ci/multi_project_pipelines.md) **(PREMIUM)** - Request [approvals](merge_request_approvals.md) from your managers **(STARTER)** - Analyze the impact of your changes with [Code Quality reports](code_quality.md) **(STARTER)** -- Manage the licenses of your dependencies with [License Management](../../application_security/license_management/index.md) **(ULTIMATE)** +- Manage the licenses of your dependencies with [License Compliance](../../application_security/license_management/index.md) **(ULTIMATE)** - Analyze your source code for vulnerabilities with [Static Application Security Testing](../../application_security/sast/index.md) **(ULTIMATE)** - Analyze your running web applications for vulnerabilities with [Dynamic Application Security Testing](../../application_security/dast/index.md) **(ULTIMATE)** - Analyze your dependencies for vulnerabilities with [Dependency Scanning](../../application_security/dependency_scanning/index.md) **(ULTIMATE)** - Analyze your Docker images for vulnerabilities with [Container Scanning](../../application_security/container_scanning/index.md) **(ULTIMATE)** - Determine the performance impact of changes with [Browser Performance Testing](#browser-performance-testing-premium) **(PREMIUM)** +- Specify merge order dependencies with [Cross-project Merge Request Dependencies](#cross-project-merge-request-dependencies-premium) **(PREMIUM)** ## Use cases @@ -56,7 +57,7 @@ A. Consider you are a software developer working in a team: 1. You gather feedback from your team 1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) **(STARTER)** 1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD -1. You avoid using dependencies whose license is not compatible with your project with [License Management reports](license_management.md) **(ULTIMATE)** +1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](license_management.md) **(ULTIMATE)** 1. You request the [approval](#merge-request-approvals-starter) from your manager 1. Your manager pushes a commit with their final review, [approves the merge request](merge_request_approvals.md), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD @@ -155,13 +156,13 @@ and remember to merge the request manually. [Learn more about merging when pipeline succeeds.](merge_when_pipeline_succeeds.md) -## Resolve discussion comments in merge requests reviews +## Resolve threads in merge requests reviews Keep track of the progress during a code review with resolving comments. Resolving comments prevents you from forgetting to address feedback and lets -you hide discussions that are no longer relevant. +you hide threads that are no longer relevant. -[Read more about resolving discussion comments in merge requests reviews.](../../discussions/index.md) +[Read more about resolving threads in merge requests reviews.](../../discussions/index.md) ## Commenting on any file line in merge requests @@ -192,7 +193,7 @@ commit when merging, to allow for a neater commit history. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/18008) in GitLab 11.6. As a reviewer, you can add suggestions to change the content in -merge request discussions, and users with appropriate [permission](../../permissions.md) +merge request threads, and users with appropriate [permission](../../permissions.md) can easily apply them to the codebase directly from the UI. Read through the documentation on [Suggest changes](../../discussions/index.md#suggest-changes) to learn more. @@ -274,7 +275,8 @@ branch already exists, the patches will be applied on top of it. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26752) in GitLab 11.10. NOTE: **Note:** -Git push options are only available with Git 2.10 or newer. +Git push options are only available with Git 2.10 or newer. With Git older than 2.18 +`git push --push-option=...` should be used instead of `git push -o ...`. GitLab supports using [Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) @@ -284,6 +286,9 @@ as pushing changes: - Create a new merge request for the pushed branch. - Set the target of the merge request to a particular branch. - Set the merge request to merge when its pipeline succeeds. +- Set the merge request to remove the source branch when it's merged. +- Set the title of the merge request to a particular title. +- Set the description of the merge request to a particular description. ### Create a new merge request using git push options @@ -327,6 +332,49 @@ pipeline succeeds at the same time using a `-o` flag per push option: git push -o merge_request.create -o merge_request.merge_when_pipeline_succeeds ``` +### Set removing the source branch using git push options + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/64320) in GitLab 12.2. + +To set an existing merge request to remove the source branch when the +merge request is merged, the +`merge_request.remove_source_branch` push option can be used: + +```sh +git push -o merge_request.remove_source_branch +``` + +You can also use this push option in addition to the +`merge_request.create` push option. + +### Set merge request title using git push options + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/64320) in GitLab 12.2. + +To set the title of an existing merge request, use +the `merge_request.title` push option: + +```sh +git push -o merge_request.title="The title I want" +``` + +You can also use this push option in addition to the +`merge_request.create` push option. + +### Set merge request description using git push options + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/64320) in GitLab 12.2. + +To set the description of an existing merge request, use +the `merge_request.description` push option: + +```sh +git push -o merge_request.description="The description I want" +``` + +You can also use this push option in addition to the +`merge_request.create` push option. + ## Find the merge request that introduced a change > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2383) in GitLab 10.5. @@ -364,6 +412,11 @@ have been marked as a **Work In Progress**. [Learn more about setting a merge request as "Work In Progress".](work_in_progress_merge_requests.md) +## Merge Requests for Confidential Issues + +Create [merge requests to resolve confidential issues](../issues/confidential_issues.md#merge-requests-for-confidential-issues) +for preventing leakage or early release of sentive data through regular merge requests. + ## Merge request approvals **(STARTER)** > Included in [GitLab Starter][products]. @@ -385,6 +438,17 @@ can show the Code Climate report right in the merge request widget area. [Read more about Code Quality reports.](code_quality.md) +## Metrics Reports **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium][products] 11.10. +Requires GitLab Runner 11.10 and above. + +If you are using [GitLab CI][ci], you can configure your job to output custom +metrics and GitLab will display the Metrics Report on the merge request so +that it's fast and easy to identify changes to important metrics. + +[Read more about Metrics Report](../../../ci/metrics_reports.md). + ## Browser Performance Testing **(PREMIUM)** > Introduced in [GitLab Premium][products] 10.3. @@ -395,6 +459,21 @@ GitLab runs the [Sitespeed.io container][sitespeed-container] and displays the d [Read more about Browser Performance Testing.](browser_performance_testing.md) +## Cross-project Merge Request Dependencies **(PREMIUM)** + +> Introduced in [GitLab Premium][products] 12.2. + +A single logical change may be split across several merge requests, across +several projects. When this happens, the order in which MRs are merged is +important. + +GitLab allows you to specify that a merge request depends on other MRs. With +this relationship in place, the merge request cannot be merged until all of its +dependencies have also been merged, helping to maintain the consistency of a +single logical change. + +[Read more about cross-project merge request dependencies.](merge_request_dependencies.md) + ## Security reports **(ULTIMATE)** GitLab can scan and report any vulnerabilities found in your project. @@ -409,15 +488,6 @@ without having to check the entire job log. [Read more about JUnit test reports](../../../ci/junit_test_reports.md). -## Live preview with Review Apps - -If you configured [Review Apps](https://about.gitlab.com/features/review-apps/) for your project, -you can preview the changes submitted to a feature-branch through a merge request -in a per-branch basis. No need to checkout the branch, install and preview locally; -all your changes will be available to preview by anyone with the Review Apps link. - -[Read more about Review Apps.](../../../ci/review_apps/index.md) - ## Merge request diff file navigation When reviewing changes in the **Changes** tab the diff can be navigated using @@ -427,6 +497,15 @@ list.  +### Incrementally expand merge request diffs + +By default, the diff shows only the parts of a file which are changed. +To view more unchanged lines above or below a change click on the +**Expand up** or **Expand down** icons. You can also click on **Show all lines** +to expand the entire file. + + + ## Ignore whitespace changes in Merge Request diff view If you click the **Hide whitespace changes** button, you can see the diff diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index 0f392676316..a0432414bcf 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -44,7 +44,7 @@ To edit the merge request approvals: 1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. -  +  1. Click **Edit**. 1. Search for users or groups that will be [eligible to approve](#eligible-approvers) @@ -54,7 +54,7 @@ To edit the merge request approvals: box. Note: the minimum can be 0. 1. Click **Update approvers**. -  +  The steps above are the minimum required to get approvals working in your merge requests, but there are a couple more options available that might be @@ -83,7 +83,7 @@ request approval rules: 1. Give the approval rule a name that describes the set of approvers selected. 1. Click **Add approvers** to submit the new rule. -  +  ## Multiple approval rules **(PREMIUM)** @@ -144,14 +144,17 @@ the following is possible: - If the required number of approvals has _not_ been yet met, they can approve it by clicking the displayed **Approve** button. -  + +  + - If the required number of approvals has already been met, they can still approve it by clicking the displayed **Approve additionally** button. -  + +  - **They have already approved this merge request**: They can remove their approval. -  +  NOTE: **Note:** The merge request author is only allowed to approve their own merge request @@ -159,10 +162,10 @@ if [**Prevent author approval**](#allowing-merge-request-authors-to-approve-thei For a given merge request, if the approval restrictions have been satisfied, the merge request is unblocked and can be merged. -Note, that meeting the required number of approvals is a necessary, but not +Note that meeting the required number of approvals is a necessary, but not sufficient condition for unblocking a merge request from being merged. There are other conditions that may block it, such as merge conflicts, -[pending discussions](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved) +[pending discussions](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-threads-are-resolved) or a [failed CI/CD pipeline](merge_when_pipeline_succeeds.md). ## Code Owners approvals **(PREMIUM)** @@ -203,7 +206,7 @@ First, you have to enable this option in the project's settings: 1. Tick the "Can override approvers and approvals required per merge request" checkbox -  +  1. Click **Save changes** @@ -226,12 +229,6 @@ The default approval settings can now be overridden when creating a in the **No. approvals required** box. 1. Click **Update approvers**. -There are however some restrictions: - -- The amount of required approvals, if changed, must be greater than the default - set at the project level. This ensures that you're not forced to adjust settings - when someone is unavailable for approval, yet the process is still enforced. - NOTE: **Note:** If you are contributing to a forked project, things are a little different. Read what happens when the @@ -239,14 +236,9 @@ Read what happens when the ## Overriding merge request approvals default settings **(PREMIUM)** -In GitLab Premium, when the approval rules are [set at the project level](#editing-approvals-premium), and -**Can override approvers and approvals required per merge request** is checked, there are a few more -restrictions (compared to [GitLab Starter](#overriding-the-merge-request-approvals-default-settings)): - -- Approval rules can be added to an MR with no restriction. -- For project sourced approval rules, editing and removing approvers is not allowed. -- The approvals required of all approval rules is configurable, but if a rule is backed by a project rule, then it is restricted - to the minimum approvals required set in the project's corresponding rule. +In GitLab Premium, when the approval rules are [set at the project level](#editing-approvals-premium), +and **Can override approvers and approvals required per merge request** is checked, +approval rules can be added to an MR with no restriction. ## Resetting approvals on push @@ -259,7 +251,7 @@ new commits are pushed to the source branch of the merge request: 1. Tick the "Remove all approvals in a merge request when new commits are pushed to its source branch" checkbox -  +  1. Click **Save changes** @@ -331,6 +323,16 @@ the dropdown) `approver` and select the user.  +## Security approvals in merge requests **(ULTIMATE)** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 12.2. + +Merge Request Approvals can be configured to require approval from a member +of your security team when a vulnerability would be introduced by a merge request. + +For more information, see +[Security approvals in merge requests](../../application_security/index.md#security-approvals-in-merge-requests-ultimate). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/merge_requests/merge_request_dependencies.md b/doc/user/project/merge_requests/merge_request_dependencies.md new file mode 100644 index 00000000000..b30e24b2386 --- /dev/null +++ b/doc/user/project/merge_requests/merge_request_dependencies.md @@ -0,0 +1,142 @@ +--- +type: reference, concepts +--- + +# Cross-project Merge Request dependencies **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9688) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. + +Cross-project merge request dependencies allows a required order of merging +between merge requests in different projects to be expressed. If a +merge request "depends on" another, then it cannot be merged until its +dependency is itself merged. + +NOTE: **Note:** +Merge requests dependencies are a **PREMIUM** feature, but this restriction is +only enforced for the dependent merge request. A merge request in a **CORE** or +**STARTER** project can be a dependency of a **PREMIUM** merge request, but not +vice-versa. + +NOTE: **Note:** +A merge request can only depend on merge requests in a different project. Two +merge requests in the same project cannot depend on each other. + +## Use cases + +- Ensure changes to a library are merged before changes to a project that + imports the library. +- Prevent a documentation-only merge request from being merged before the merge request + implementing the feature to be documented. +- Require an merge request updating a permissions matrix to be merged before merging an + merge request from someone who hasn't yet been granted permissions. + +It is common for a single logical change to span several merge requests, spread +out across multiple projects, and the order in which they are merged can be +significant. + +For example, given a project `mycorp/awesome-project` that imports a library +at `myfriend/awesome-lib`, adding a feature in `awesome-project` may **also** +require changes to `awesome-lib`, and so necessitate two merge requests. Merging +the `awesome-project` merge request before the `awesome-lib` one would +break the `master`branch. + +The `awesome-project` merge request could be [marked as +WIP](work_in_progress_merge_requests.md), +and the reason for the WIP stated included in the comments. However, this +requires the state of the `awesome-lib` merge request to be manually +tracked, and doesn't scale well if the `awesome-project` merge request +depends on changes to **several** other projects. + +By making the `awesome-project` merge request depend on the +`awesome-lib` merge request instead, this relationship is +automatically tracked by GitLab, and the WIP state can be used to +communicate the readiness of the code in each individual merge request +instead. + +## Configuration + +To continue the above example, you can configure a dependency when creating the +new merge request in `awesome-project` (or by editing it, if it already exists). +The dependency needs to be configured on the **dependent** merge +request. There is a "Cross-project dependencies" section in the form: + + + +Anyone who can edit a merge request can change the list of dependencies. + +New dependencies can be added by reference, or by URL. To remove a dependency, +press the **X** by its reference. + +As dependencies are specified across projects, it's possible that someone else +has added a dependency for a merge request in a project you don't have access to. +These are shown as a simple count: + + + +If necessary, you can remove all the dependencies like this by pressing the +**X**, just as you would for a single, visible dependency. + +Once you're finished, press the **Save changes** button to submit the request, +or **Cancel** to return without making any changes. + +The list of configured dependencies, and the status of each one, is shown in the +merge request widget: + + + +Until all dependencies have, themselves, been merged, the **Merge** +button will be disabled for the dependent merge request. In +particular, note that **closed merge requests** still prevent their +dependents from being merged - it is impossible to automatically +determine whether the dependency expressed by a closed merge request +has been satisfied in some other way or not. + +If a merge request has been closed **and** the dependency is no longer relevant, +it must be removed as a dependency, following the instructions above, before +merge. + +## Limitations + +- API support: [gitlab-ee#12551](https://gitlab.com/gitlab-org/gitlab-ee/issues/12551) +- Dependencies are not preserved across project export/import: [gitlab-ee#12549](https://gitlab.com/gitlab-org/gitlab-ee/issues/12549) +- Complex merge order dependencies are not supported: [gitlab-ee#11393](https://gitlab.com/gitlab-org/gitlab-ee/issues/11393) + +The last item merits a little more explanation. Dependencies between merge +requests can be described as a graph of relationships. The simplest possible +graph has one merge request that depends upon another: + +```mermaid +graph LR; + myfriend/awesome-lib!10-->mycorp/awesome-project!100; +``` + +A more complex (and still supported) graph might have one merge request that +directly depends upon several others: + +```mermaid +graph LR; + myfriend/awesome-lib!10-->mycorp/awesome-project!100; + herfriend/another-lib!1-->mycorp/awesome-project!100; +``` + +Several different merge requests can also directly depend upon the +same merge request: + +```mermaid +graph LR; + herfriend/another-lib!1-->myfriend/awesome-lib!10; + herfriend/another-lib!1-->mycorp/awesome-project!100; +``` + +What is **not** supported is a "deep", or "nested" graph of dependencies, e.g.: + +```mermaid +graph LR; + herfriend/another-lib!1-->myfriend/awesome-lib!10; + myfriend/awesome-lib!10-->mycorp/awesome-project!100; +``` + +In this example, `myfriend/awesome-lib!10` depends on `herfriend/another-lib!1`, +and is itself a dependent of `mycorp/awesome-project!100`. This means that +`myfriend/awesome-lib!10` becomes an **indirect** dependency of +`mycorp/awesome-project!100`, which is not yet supported. diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index 0dd60d84c42..50a4e514d49 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -36,7 +36,7 @@ changes to be reviewed. ## Only allow merge requests to be merged if the pipeline succeeds You can prevent merge requests from being merged if their pipeline did not succeed -or if there are discussions to be resolved. +or if there are threads to be resolved. Navigate to your project's settings page and expand the **Merge requests** section. In the **Merge checks** subsection, select the **Pipelines must succeed** check diff --git a/doc/user/project/merge_requests/work_in_progress_merge_requests.md b/doc/user/project/merge_requests/work_in_progress_merge_requests.md index 142178ba241..ea59644fce6 100644 --- a/doc/user/project/merge_requests/work_in_progress_merge_requests.md +++ b/doc/user/project/merge_requests/work_in_progress_merge_requests.md @@ -5,7 +5,7 @@ type: reference, concepts # "Work In Progress" merge requests If a merge request is not yet ready to be merged, perhaps due to continued development -or open discussions, you can prevent it from being accepted before it's ready by flagging +or open threads, you can prevent it from being accepted before it's ready by flagging it as a **Work In Progress**. This will disable the "Merge" button, preventing it from being merged, and it will stay disabled until the "WIP" flag has been removed. @@ -19,7 +19,7 @@ There are several ways to flag a merge request as a Work In Progress: **Start the title with WIP:**, under the title box, when editing the merge request's description will have the same effect. - Add the `/wip` [quick action](../quick_actions.md#quick-actions-for-issues-and-merge-requests) - in a discussion comment in the merge request. This is a toggle, and can be repeated + in a comment in the merge request. This is a toggle, and can be repeated to change the status back. Note that any other text in the comment will be discarded. - Add "wip" or "WIP" to the start of a commit message targeting the merge request's source branch. This is not a toggle, and doing it again in another commit will have @@ -34,7 +34,7 @@ Similar to above, when a Merge Request is ready to be merged, you can remove the **Remove the WIP: prefix from the title**, under the title box, when editing the merge request's description, will have the same effect. - Add the `/wip` [quick action](../quick_actions.md#quick-actions-for-issues-and-merge-requests) - in a discussion comment in the merge request. This is a toggle, and can be repeated + in a comment in the merge request. This is a toggle, and can be repeated to change the status back. Note that any other text in the comment will be discarded. - Click on the **Resolve WIP status** button near the bottom of the merge request description, next to the "Merge" button (see [image above](#work-in-progress-merge-requests)). diff --git a/doc/user/project/milestones/burndown_charts.md b/doc/user/project/milestones/burndown_charts.md index 84e08b4eeb8..d4dd9c7d4c3 100644 --- a/doc/user/project/milestones/burndown_charts.md +++ b/doc/user/project/milestones/burndown_charts.md @@ -1,6 +1,9 @@ +--- +type: reference +--- + # Burndown Charts **(STARTER)** -> **Notes:** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1540) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.1 for project milestones. > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5354) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.8 for group milestones. > - [Added](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6495) to [GitLab Starter](https://about.gitlab.com/pricing/) 11.2 for group milestones. @@ -20,7 +23,8 @@ yourself to have the same sense of progress. GitLab Starter plots it for you and presents it in a clear and beautiful chart. -For an overview, check the video demonstration on [Mapping Work Versus Time, With Burndown Charts](https://about.gitlab.com/2017/04/25/mapping-work-to-do-versus-time-with-burndown-charts/). +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, check the video demonstration on [Mapping work versus time with Burndown Charts](https://www.youtube.com/watch?v=zJU2MuRChzs). ## Use cases @@ -68,3 +72,15 @@ The Burndown Chart can also be toggled to display the cumulative open issue weight for a given day. When using this feature, make sure issue weights have been properly assigned, since an open issue with no weight adds zero to the cumulative value. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index b0745b1e2c5..453983fa882 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -1,3 +1,7 @@ +--- +type: index, reference +--- + # Milestones ## Overview @@ -14,12 +18,12 @@ the start and end of your Agile sprint. Set the milestone title to the name of your Agile sprint, such as `November 2018 sprint`. Add an issue to your Agile sprint by associating -the milestone to the issue. +the milestone to the issue. ## Milestones as releases Milestones can be used as releases. -Set the milestone due date to represent the release date of your release. +Set the milestone due date to represent the release date of your release. (And leave the milestone start date blank.) Set the milestone title to the version of your release, such as `Version 9.4`. @@ -145,3 +149,15 @@ The milestone sidebar on the milestone view shows the following: For project milestones only, the milestone sidebar shows the total issue weight of all issues that have the milestone assigned.  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index d35a8568394..8606d92f20c 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -6,8 +6,6 @@ GitLab 8.12 has a completely redesigned [job permissions] system. You can find all discussion and all our concerns when choosing the current approach in issue [#18994](https://gitlab.com/gitlab-org/gitlab-ce/issues/18994). ---- - Jobs permissions should be tightly integrated with the permissions of a user who is triggering a job. @@ -30,13 +28,13 @@ The reasons to do it like that are: With the new behavior, any job that is triggered by the user, is also marked with their read permissions. When a user does a `git push` or changes files through the web UI, a new pipeline will be usually created. This pipeline will be marked -as created be the pusher (local push or via the UI) and any job created in this +as created by the pusher (local push or via the UI) and any job created in this pipeline will have the read permissions of the pusher but not write permissions. This allows us to make it really easy to evaluate the access for all projects that have [Git submodules][gitsub] or are using container images that the pusher -would have access too. **The permission is granted only for time that job is -running. The access is revoked after the job is finished.** +would have access too. **The permission is granted only for the time that the job +is running. The access is revoked after the job is finished.** ## Types of users @@ -76,7 +74,7 @@ We try to make sure that this token doesn't leak by: 1. Securing all API endpoints to not expose the job token. 1. Masking the job token from job logs. -1. Allowing to use the job token **only** when job is running. +1. Granting permissions to the job token **only** when the job is running. However, this brings a question about the Runners security. To make sure that this token doesn't leak, you should also make sure that you configure @@ -88,13 +86,6 @@ your Runners in the most possible secure way, by avoiding the following: By using an insecure GitLab Runner configuration, you allow the rogue developers to steal the tokens of other jobs. -## Pipeline triggers - -Since 9.0 [pipeline triggers][triggers] do support the new permission model. -The new triggers do impersonate their associated user including their access -to projects and their project permissions. To migrate trigger to use new permission -model use **Take ownership**. - ## Before GitLab 8.12 In versions before GitLab 8.12, all CI jobs would use the CI Runner's token @@ -108,8 +99,6 @@ and to checkout project sources. It could also be used with the GitLab Container Registry for that project, allowing pulling and pushing Docker images from within the CI job. ---- - GitLab would create a special checkout URL like: ``` @@ -208,7 +197,7 @@ Container Registries for private projects. > **Notes:** > > - GitLab Runner versions prior to 1.8 don't incorporate the introduced changes -> for permissions. This makes the `image:` directive to not work with private +> for permissions. This makes the `image:` directive not work with private > projects automatically and it needs to be configured manually on Runner's host > with a predefined account (for example administrator's personal account with > access token created explicitly for this purpose). This issue is resolved with @@ -232,11 +221,22 @@ test: - docker run $CI_REGISTRY/group/other-project:latest ``` +### Pipeline triggers + +Since 9.0 [pipeline triggers][triggers] do support the new permission model. +The new triggers do impersonate their associated user including their access +to projects and their project permissions. + +### API + +GitLab API cannot be used via `CI_JOB_TOKEN` but there is a [proposal](https://gitlab.com/gitlab-org/gitlab-ce/issues/29566) +to support it. + [job permissions]: ../permissions.md#job-permissions [comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302 [gitsub]: ../../ci/git_submodules.md [https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols -[triggers]: ../../ci/triggers/README.md +[triggers]: ../../ci/triggers/README.md#ci-job-token [update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update [workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse [jobenv]: ../../ci/variables/README.md#predefined-environment-variables diff --git a/doc/user/project/operations/feature_flags.md b/doc/user/project/operations/feature_flags.md index fdc1e978291..75b0623e6b0 100644 --- a/doc/user/project/operations/feature_flags.md +++ b/doc/user/project/operations/feature_flags.md @@ -74,9 +74,9 @@ To define specs for each environment: 1. Navigate to your project's **Operations > Feature Flags**. 1. Click on the **New Feature Flag** button or edit an existing flag. -1. Set the status of the default [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) (`*`). This status will be used for _all_ environments. +1. Set the status of the default [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) (`*`). Choose a rollout strategy. This status and rollout strategy combination will be used for _all_ environments. 1. If you want to enable/disable the feature on a specific environment, create a new [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) and type the environment name. -1. Set the status of the additional spec. This status takes precedence over the default spec's status since we always use the most specific match available. +1. Set the status and rollout strategy of the additional spec. This status and rollout strategy combination takes precedence over the default spec since we always use the most specific match available. 1. Click **Create feature flag** or **Update feature flag**.  @@ -85,6 +85,46 @@ NOTE: **NOTE** We'd highly recommend you to use the [Environment](../../../ci/environments.md) feature in order to quickly assess which flag is enabled per environment. +## Rollout Strategy + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8240) in GitLab 12.2. + +The selected rollout strategy affects which users will experience the feature enabled. + +The status of an environment spec ultimately determines whether or not a feature is enabled at all. +For instance, a feature will always be disabled for every user if the matching environment spec has a disabled status, regardless of the chosen rollout strategy. +However, a feature will be enabled for 50% of logged-in users if the matching environment spec has an enabled status along with a **Percent rollout (logged in users)** strategy set to 50%. + +### All users + +Enables the feature for all users. + +**All users** is implemented using the Unleash [default](https://unleash.github.io/docs/activation_strategy#default) activation strategy. + +### Percent rollout (logged in users) + +**Percent rollout (logged in users)** enables the feature for a percentage of authenticated users. Set a value of 15%, for example, to enable the feature for 15% of authenticated users. + +A rollout percentage may be between 0% and 100%. + +CAUTION: **Caution:** +If this strategy is selected, then the Unleash client **must** be given a user id for the feature to be enabled. See the [Ruby example](#ruby-application-example) below. + +**Percent rollout (logged in users)** is implemented using the Unleash [gradualRolloutUserId](https://unleash.github.io/docs/activation_strategy#gradualrolloutuserid) activation strategy. + +## Target Users + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8240) in GitLab 12.2. + +A feature flag may be enabled for a list of target users. + + + +CAUTION: **Caution:** +The Unleash client **must** be given a user id for the feature to be enabled for target users. See the [Ruby example](#ruby-application-example) below. + +**Target users** is implemented using the Unleash [userWithId](https://unleash.github.io/docs/activation_strategy#userwithid) activation strategy. + ## Integrating with your application In order to use Feature Flags, you need to first @@ -175,3 +215,34 @@ func main() { log.Fatal(http.ListenAndServe(":8080", nil)) } ``` + +### Ruby application example + +Here's an example of how to integrate the feature flags in a Ruby application. + +The Unleash client is given a user id for use with a **Percent rollout (logged in users)** rollout strategy or a list of **Target Users**. + +```ruby +#!/usr/bin/env ruby + +require 'unleash' +require 'unleash/context' + +unleash = Unleash::Client.new({ + url: 'http://gitlab.com/api/v4/feature_flags/unleash/42', + app_name: 'production', + instance_id: '29QmjsW6KngPR5JNPMWx' +}) + +unleash_context = Unleash::Context.new +# Replace "123" with the id of an authenticated user. +# Note that the context's user id must be a string: +# https://unleash.github.io/docs/unleash_context +unleash_context.user_id = "123" + +if unleash.is_enabled?("my_feature_name", unleash_context) + puts "Feature enabled" +else + puts "hello, world!" +end +``` diff --git a/doc/user/project/operations/img/error_tracking_list.png b/doc/user/project/operations/img/error_tracking_list.png Binary files differindex aa0f9867fdb..194c7b440a4 100644 --- a/doc/user/project/operations/img/error_tracking_list.png +++ b/doc/user/project/operations/img/error_tracking_list.png diff --git a/doc/user/project/operations/img/external_dashboard_link.png b/doc/user/project/operations/img/external_dashboard_link.png Binary files differindex 4fb8bce7cd0..82c5e05e467 100644 --- a/doc/user/project/operations/img/external_dashboard_link.png +++ b/doc/user/project/operations/img/external_dashboard_link.png diff --git a/doc/user/project/operations/img/external_dashboard_settings.png b/doc/user/project/operations/img/external_dashboard_settings.png Binary files differindex 8dc380f01e2..e1b7fa56a1c 100644 --- a/doc/user/project/operations/img/external_dashboard_settings.png +++ b/doc/user/project/operations/img/external_dashboard_settings.png diff --git a/doc/user/project/operations/img/feature_flags_list.png b/doc/user/project/operations/img/feature_flags_list.png Binary files differindex 5313a163fec..f3e85b9ce44 100644 --- a/doc/user/project/operations/img/feature_flags_list.png +++ b/doc/user/project/operations/img/feature_flags_list.png diff --git a/doc/user/project/operations/img/specs_list.png b/doc/user/project/operations/img/specs_list.png Binary files differindex 9630c907cfc..43d069c09ce 100644 --- a/doc/user/project/operations/img/specs_list.png +++ b/doc/user/project/operations/img/specs_list.png diff --git a/doc/user/project/operations/img/target_users_v12_2.png b/doc/user/project/operations/img/target_users_v12_2.png Binary files differnew file mode 100644 index 00000000000..c88d2b7be97 --- /dev/null +++ b/doc/user/project/operations/img/target_users_v12_2.png diff --git a/doc/user/project/packages/img/npm_package_view.png b/doc/user/project/packages/img/npm_package_view.png Binary files differindex 8baf7d0ef9f..e0634718c02 100644 --- a/doc/user/project/packages/img/npm_package_view.png +++ b/doc/user/project/packages/img/npm_package_view.png diff --git a/doc/user/project/packages/maven_repository.md b/doc/user/project/packages/maven_repository.md index 27c052fb2bc..c61402afb2c 100644 --- a/doc/user/project/packages/maven_repository.md +++ b/doc/user/project/packages/maven_repository.md @@ -198,7 +198,7 @@ domain name. NOTE: **Note:** For retrieving artifacts, you can use either the [URL encoded](../../../api/README.md#namespaced-path-encoding) path of the group -(e.g., `group%2Fsubgroup`) or the group's ID (e.g., `12`). +(e.g., `group%2Fsubgroup`) or the group's ID (e.g., `12`). ### Instance level Maven endpoint @@ -279,59 +279,59 @@ shows how to create a new package each time the `master` branch is updated: Add the server section with the same id you defined in your `pom.xml` file. For example, in our case it's `gitlab-maven`: - ```xml - <settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd"> - <servers> - <server> - <id>gitlab-maven</id> - <configuration> - <httpHeaders> - <property> - <name>Job-Token</name> - <value>${env.CI_JOB_TOKEN}</value> - </property> - </httpHeaders> - </configuration> - </server> - </servers> - </settings> - ``` + ```xml + <settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd"> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> + </settings> + ``` 1. Make sure your `pom.xml` file includes the following: - ```xml - <repositories> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> - </repository> - </repositories> - <distributionManagement> - <repository> - <id>gitlab-maven</id> - <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> - </repository> - <snapshotRepository> - <id>gitlab-maven</id> - <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> - </snapshotRepository> - </distributionManagement> - ``` - - TIP: **Tip:** - You can either let Maven utilize the CI environment variables or hardcode your project's ID. + ```xml + <repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + </repositories> + <distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </snapshotRepository> + </distributionManagement> + ``` + + TIP: **Tip:** + You can either let Maven utilize the CI environment variables or hardcode your project's ID. 1. Add a `deploy` job to your `.gitlab-ci.yml` file: - ```yaml - deploy: - image: maven:3.3.9-jdk-8 - script: - - 'mvn deploy -s ci_settings.xml' - only: - - master - ``` + ```yaml + deploy: + image: maven:3.3.9-jdk-8 + script: + - 'mvn deploy -s ci_settings.xml' + only: + - master + ``` 1. Push those files to your repository. diff --git a/doc/user/project/packages/npm_registry.md b/doc/user/project/packages/npm_registry.md index 481b1ce0337..e01bac6b26f 100644 --- a/doc/user/project/packages/npm_registry.md +++ b/doc/user/project/packages/npm_registry.md @@ -49,41 +49,42 @@ Registry. ## Authenticating to the GitLab NPM Registry If a project is private or you want to upload an NPM package to GitLab, -credentials will need to be provided for authentication. Support is available -only for [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). +credentials will need to be provided for authentication. Support is available for [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow) or [personal access tokens](../../profile/personal_access_tokens.md). -CAUTION: **2FA not supported:** -Authentication for personal access tokens is not yet supported -([#9140](https://gitlab.com/gitlab-org/gitlab-ee/issues/9140)). If you have 2FA -enabled, you won't be able to authenticate to the GitLab NPM Registry. +CAUTION: **2FA is only supported with personal access tokens:** +If you have 2FA enabled, you need to use a [personal access token](../../profile/personal_access_tokens.md) with OAuth headers. Standard OAuth tokens won't be able to authenticate to the GitLab NPM Registry. ### Authenticating with an OAuth token -To authenticate with an [OAuth token](../../../api/oauth2.md#resource-owner-password-credentials-flow), -add a corresponding section to your `.npmrc` file: +To authenticate with an [OAuth token](../../../api/oauth2.md#resource-owner-password-credentials-flow) +or [personal access token](../../profile/personal_access_tokens.md), add a corresponding section to your `.npmrc` file: ```ini ; Set URL for your scoped packages. ; For example package with name `@foo/bar` will use this URL for download @foo:registry=https://gitlab.com/api/v4/packages/npm/ -; Add the OAuth token for the scoped packages URL. This will allow you to download +; Add the token for the scoped packages URL. This will allow you to download ; `@foo/` packages from private projects. -//gitlab.com/api/v4/packages/npm/:_authToken=<your_oauth_token> +//gitlab.com/api/v4/packages/npm/:_authToken=<your_token> -; Add OAuth token for uploading to the registry. Replace <your_project_id> +; Add token for uploading to the registry. Replace <your_project_id> ; with the project you want your package to be uploaded to. -//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_oauth_token> +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_token> ``` Replace `<your_project_id>` with your project ID which can be found on the home page -of your project and `<your_oauth_token>` with your OAuth token. +of your project and `<your_token>` with your OAuth or personal access token. If you have a self-hosted GitLab installation, replace `gitlab.com` with your domain name. You should now be able to download and upload NPM packages to your project. +NOTE: **Note:** +If you encounter an error message with [Yarn](https://yarnpkg.com/en/), see the +[troubleshooting section](#troubleshooting). + ## Uploading packages Before you will be able to upload a package, you need to specify the registry @@ -119,3 +120,29 @@ a given scope, you will receive a `403 Forbidden!` error. If you upload a package with a same name and version twice, GitLab will show both packages in the UI, but the GitLab NPM Registry will expose the most recent one as it supports only one package per version for `npm install`. + +## Troubleshooting + +### Error running yarn with NPM registry + +If you are using [yarn](https://yarnpkg.com/en/) with the NPM registry, you may get +an error message like: + +```sh +yarn install v1.15.2 +warning package.json: No license field +info No lockfile found. +warning XXX: No license field +[1/4] 🔍 Resolving packages... +[2/4] 🚚 Fetching packages... +error An unexpected error occurred: "https://gitlab.com/api/v4/projects/XXX/packages/npm/XXX/XXX/-/XXX/XXX-X.X.X.tgz: Request failed \"404 Not Found\"". +info If you think this is a bug, please open a bug report with the information provided in "/Users/XXX/gitlab-migration/module-util/yarn-error.log". +info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command +``` + +In this case, try adding this to your `.npmrc` file (and replace `<your_oauth_token>` +with your with your OAuth or personal access token): + +```text +//gitlab.com/api/v4/projects/:_authToken=<your_oauth_token> +``` diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md new file mode 100644 index 00000000000..3b80370d4a9 --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md @@ -0,0 +1,95 @@ +--- +type: concepts +--- + +# DNS records overview + +_Read this document for a brief overview of DNS records in the scope +of GitLab Pages, for beginners in web development._ + +A Domain Name System (DNS) web service routes visitors to websites +by translating domain names (such as `www.example.com`) into the +numeric IP addresses (such as `192.0.2.1`) that computers use to +connect to each other. + +A DNS record is created to point a (sub)domain to a certain location, +which can be an IP address or another domain. In case you want to use +GitLab Pages with your own (sub)domain, you need to access your domain's +registrar control panel to add a DNS record pointing it back to your +GitLab Pages site. + +Note that **how to** add DNS records depends on which server your domain +is hosted on. Every control panel has its own place to do it. If you are +not an admin of your domain, and don't have access to your registrar, +you'll need to ask for the technical support of your hosting service +to do it for you. + +To help you out, we've gathered some instructions on how to do that +for the most popular hosting services: + +- [Amazon](http://docs.aws.amazon.com/gettingstarted/latest/swh/getting-started-configure-route53.html) +- [Bluehost](https://my.bluehost.com/cgi/help/559) +- [CloudFlare](https://support.cloudflare.com/hc/en-us/articles/200169096-How-do-I-add-A-records-) +- [cPanel](https://documentation.cpanel.net/display/ALD/Edit+DNS+Zone) +- [DreamHost](https://help.dreamhost.com/hc/en-us/articles/215414867-How-do-I-add-custom-DNS-records-) +- [Go Daddy](https://www.godaddy.com/help/add-an-a-record-19238) +- [Hostgator](http://support.hostgator.com/articles/changing-dns-records) +- [Inmotion hosting](https://my.bluehost.com/cgi/help/559) +- [Media Temple](https://mediatemple.net/community/products/dv/204403794/how-can-i-change-the-dns-records-for-my-domain) +- [Microsoft](https://msdn.microsoft.com/en-us/library/bb727018.aspx) + +If your hosting service is not listed above, you can just try to +search the web for `how to add dns record on <my hosting service>`. + +## `A` record + +A DNS A record maps a host to an IPv4 IP address. +It points a root domain as `example.com` to the host's IP address as +`192.192.192.192`. + +Example: + +- `example.com` => `A` => `192.192.192.192` + +## CNAME record + +CNAME records define an alias for canonical name for your server (one defined +by an A record). It points a subdomain to another domain. + +Example: + +- `www` => `CNAME` => `example.com` + +This way, visitors visiting `www.example.com` will be redirected to +`example.com`. + +## MX record + +MX records are used to define the mail exchanges that are used for the domain. +This helps email messages arrive at your mail server correctly. + +Example: + +- `MX` => `mail.example.com` + +Then you can register emails for `users@mail.example.com`. + +## TXT record + +A `TXT` record can associate arbitrary text with a host or other name. A common +use is for site verification. + +Example: + +- `example.com`=> TXT => `"google-site-verification=6P08Ow5E-8Q0m6vQ7FMAqAYIDprkVV8fUf_7hZ4Qvc8"` + +This way, you can verify the ownership for that domain name. + +## All combined + +You can have one DNS record or more than one combined: + +- `example.com` => `A` => `192.192.192.192` +- `www` => `CNAME` => `example.com` +- `MX` => `mail.example.com` +- `example.com`=> TXT => `"google-site-verification=6P08Ow5E-8Q0m6vQ7FMAqAYIDprkVV8fUf_7hZ4Qvc8"`
\ No newline at end of file diff --git a/doc/user/project/pages/img/add_certificate_to_pages.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/add_certificate_to_pages.png Binary files differindex d92a981dc60..d92a981dc60 100644 --- a/doc/user/project/pages/img/add_certificate_to_pages.png +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/add_certificate_to_pages.png diff --git a/doc/user/project/pages/img/dns_add_new_a_record_example_updated_2018.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/dns_add_new_a_record_example_updated_2018.png Binary files differindex 0150329d4b2..0150329d4b2 100644 --- a/doc/user/project/pages/img/dns_add_new_a_record_example_updated_2018.png +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/dns_add_new_a_record_example_updated_2018.png diff --git a/doc/user/project/pages/img/dns_cname_record_example.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/dns_cname_record_example.png Binary files differindex 43d1a838544..43d1a838544 100644 --- a/doc/user/project/pages/img/dns_cname_record_example.png +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/dns_cname_record_example.png diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/img/get_domain_verification_code_v12_0.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/get_domain_verification_code_v12_0.png Binary files differnew file mode 100644 index 00000000000..3b93572a8f1 --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/get_domain_verification_code_v12_0.png diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/img/lets_encrypt_integration_v12_1.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/lets_encrypt_integration_v12_1.png Binary files differnew file mode 100644 index 00000000000..5eab04a61a9 --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/lets_encrypt_integration_v12_1.png diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/img/retry_domain_verification_v12_0.png b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/retry_domain_verification_v12_0.png Binary files differnew file mode 100644 index 00000000000..5f2f100e883 --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/img/retry_domain_verification_v12_0.png diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md new file mode 100644 index 00000000000..1821d954af3 --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -0,0 +1,276 @@ +--- +last_updated: 2019-07-04 +type: reference, howto +disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pages/getting_started_part_three.html' +--- + +# Custom domains and SSL/TLS Certificates + +Setting up GitLab Pages with custom domains, and adding SSL/TLS certificates to them, are optional features of GitLab Pages. + +To use one or more custom domain names with your Pages site, you can: + +- Add a [custom **root domain** or a **subdomain**](#set-up-pages-with-a-custom-domain). +- Add [SSL/TLS certification](#adding-an-ssltls-certificate-to-pages). + +## Set up Pages with a custom domain + +To set up Pages with a custom domain name, read the requirements +and steps below. + +### Requirements + +- A GitLab Pages website up and running, served under the default Pages domain + (`*.gitlab.io`, for GitLab.com). +- A custom domain name `example.com` or subdomain `subdomain.example.com`. +- Access to your domain's server control panel to set up DNS records: + - A DNS A or CNAME record poiting your domain to GitLab Pages server. + - A DNS TXT record to verify your domain's ownership. + +### Steps + +Follow the steps below to add your custom domain to Pages. See also +this document for an [overview on DNS records](dns_concepts.md). + +#### 1. Add a custom domain to Pages + +Navigate to your project's **Setting > Pages** and click **+ New domain** +to add your custom domain to GitLab Pages. You can choose whether to: + +- Add an [SSL/TLS certificate](#adding-an-ssltls-certificate-to-pages). +- Leave it blank (it can be added later). + +Click **Create New Domain**. + + + +#### 2. Get the verification code + +Once you have added a new domain to Pages, the verification code will be prompted to you. Copy the values from GitLab and paste them in your domain's control panel as a TXT record on the next step. + + + +#### 3. Set up DNS records for Pages + +Read this document for an [overview of DNS records for Pages](dns_concepts.md). +If you're familiar with the subject, follow the instructions below +according to the type of domain you want to use with your Pages site: + +- [For root domains](#for-root-domains), `example.com`. +- [For subdomains](#for-subdomains), `subdomain.example.com`. +- [For both](#for-both-root-and-subdomains). + +##### For root domains + +Root domains (`example.com`) require: + +- A [DNS A record](dns_concepts.md#a-record) pointing your domain to the Pages server. +- A [TXT record](dns_concepts.md#txt-record) to verify your domain's ownership. + +| From | DNS Record | To | +| ---- | ---------- | -- | +| example.com | A | 35.185.44.232 | +| _gitlab-pages-verification-code.example.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | + +For projects on GitLab.com, this IP is `35.185.44.232`. +For projects living in other GitLab instances (CE or EE), please contact +your sysadmin asking for this information (which IP address is Pages +server running on your instance). + + + +CAUTION: **Caution:** +Note that if you use your root domain for your GitLab Pages website +**only**, and if your domain registrar supports this feature, you can +add a DNS apex `CNAME` record instead of an `A` record. The main +advantage of doing so is that when GitLab Pages IP on GitLab.com +changes for whatever reason, you don't need to update your `A` record. +There may be a few exceptions, but **this method is not recommended** +as it most likely won't work if you set an [`MX` record](dns_concepts.md#mx-record) for your root domain. + +##### For subdomains + +Subdomains (`subdomain.example.com`) require: + +- A DNS [CNAME record](dns_concepts.md#cname-record) record pointing your subdomain to the Pages server. +- A DNS [TXT record](dns_concepts.md#txt-record) to verify your domain's ownership. + +| From | DNS Record | To | +| ---- | ---------- | -- | +| subdomain.example.com | CNAME | namespace.gitlab.io | +| _gitlab-pages-verification-code.subdomain.example.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | + +Note that, whether it's a user or a project website, the `CNAME` +should point to your Pages domain (`namespace.gitlab.io`), +without any `/project-name`. + + + +##### For both root and subdomains + +There are a few cases where you need point both subdomain and root +domain to the same website, for instance, `example.com` and `www.example.com`. + +They require: + +- A DNS A record for the domain. +- A DNS CNAME record for the subdomain. +- A DNS TXT record for each. + +| From | DNS Record | To | +| ---- | ---------- | -- | +| example.com | A | 35.185.44.232 | +| _gitlab-pages-verification-code.example.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | +|---+---| +| www.example.com | CNAME | namespace.gitlab.io | +| _gitlab-pages-verification-code.www.example.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | + +If you're using CloudFlare, check +[Redirecting `www.domain.com` to `domain.com` with Cloudflare](#redirecting-wwwdomaincom-to-domaincom-with-cloudflare). + +> **Notes**: +> +> - **Do not** use a CNAME record if you want to point your + `domain.com` to your GitLab Pages site. Use an `A` record instead. +> - **Do not** add any special chars after the default Pages + domain. E.g., don't point `subdomain.domain.com` to + or `namespace.gitlab.io/`. Some domain hosting providers may request a trailling dot (`namespace.gitlab.io.`), though. +> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017. +> - GitLab Pages IP on GitLab.com [has changed](https://about.gitlab.com/2018/07/19/gcp-move-update/#gitlab-pages-and-custom-domains) + from `52.167.214.135` to `35.185.44.232` in 2018. + +#### 4. Verify the domain's ownership + +Once you have added all the DNS records: + +1. Go back at your project's **Settings > Pages**. +1. Locate your domain name and click **Details**. +1. Click the **Retry verification** button to activate your new domain. + + + +As soon as your domain becomes active, your website will be available +through your domain name. + +CAUTION: **Caution:** +Considering GitLab instances with domain verification enabled, +if the domain cannot be verified for 7 days, it will be removed +from the GitLab project. + +> **Notes:** +> +> - Domain verification is **required for GitLab.com users**; + for GitLab self-managed instances, your GitLab administrator has the option + to [disabled custom domain verification](../../../../administration/pages/index.md#custom-domain-verification). +> - [DNS propagation may take some time (up to 24h)](http://www.inmotionhosting.com/support/domain-names/dns-nameserver-changes/domain-names-dns-changes), + although it's usually a matter of minutes to complete. Until it does, verification + will fail and attempts to visit your domain will respond with a 404. +> - Once your domain has been verified, leave the verification record + in place: your domain will be periodically reverified, and may be + disabled if the record is removed. + +### Adding more domain aliases + +You can add more than one alias (custom domains and subdomains) to the same project. +An alias can be understood as having many doors leading to the same room. + +All the aliases you've set to your site will be listed on **Setting > Pages**. +From that page, you can view, add, and remove them. + +### Redirecting `www.domain.com` to `domain.com` with Cloudflare + +If you use Cloudflare, you can redirect `www` to `domain.com` +without adding both `www.domain.com` and `domain.com` to GitLab. + +To do so, you can use Cloudflare's page rules associated to a +CNAME record to redirect `www.domain.com` to `domain.com`. You +can use the following setup: + +1. In Cloudflare, create a DNS `A` record pointing `domain.com` to `35.185.44.232`. +1. In GitLab, add the domain to GitLab Pages and get the verification code. +1. In Cloudflare, create a DNS `TXT` record to verify your domain. +1. In GitLab, verify your domain. +1. In Cloudflare, create a DNS `CNAME` record pointing `www` to `domain.com`. +1. In Cloudflare, add a Page Rule pointing `www.domain.com` to `domain.com`: + - Navigate to your domain's dashboard and click **Page Rules** + on the top nav. + - Click **Create Page Rule**. + - Enter the domain `www.domain.com` and click **+ Add a Setting**. + - From the dropdown menu, choose **Forwarding URL**, then select the + status code **301 - Permanent Redirect**. + - Enter the destination URL `https://domain.com`. + +## Adding an SSL/TLS certificate to Pages + +Read this document for an [overview on SSL/TLS certification](ssl_tls_concepts.md). + +To secure your custom domain with GitLab Pages you can opt by: + +- Using the [Let's Encrypt integration with GitLab Pages](lets_encrypt_integration.md), + which automatically obtains and renews SSL certificates + for your Pages domains. +- Manually adding SSL/TLS certificates to GitLab Pages websites + by following the steps below. + +### Requirements + +- A GitLab Pages website up and running accessible via a custom domain. +- **A PEM certificate**: it is the certificate generated by the CA, + which needs to be added to the field **Certificate (PEM)**. +- **An [intermediate certificate](https://en.wikipedia.org/wiki/Intermediate_certificate_authority)**: (aka "root certificate"), it is + the part of the encryption keychain that identifies the CA. + Usually it's combined with the PEM certificate, but there are + some cases in which you need to add them manually. + [CloudFlare certs](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) + are one of these cases. +- **A private key**, it's an encrypted key which validates + your PEM against your domain. + +### Steps + +- To add the certificate at the time you add a new domain, go to your + project's **Settings > Pages > New Domain**, add the domain name and the certificate. +- To add the certificate to a domain previously added, go to your + project's **Settings > Pages**, locate your domain name, click **Details** and **Edit** to add the certificate. + + + +1. Add the PEM certificate to its corresponding field. +1. If your certificate is missing its intermediate, copy + and paste the root certificate (usually available from your CA website) + and paste it in the [same field as your PEM certificate](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/), + just jumping a line between them. +1. Copy your private key and paste it in the last field. + +NOTE: **Note:** +**Do not** open certificates or encryption keys in +regular text editors. Always use code editors (such as +Sublime Text, Atom, Dreamweaver, Brackets, etc). + +## Force HTTPS for GitLab Pages websites + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28857) in GitLab 10.7. + +To make your website's visitors even more secure, you can choose to +force HTTPS for GitLab Pages. By doing so, all attempts to visit your +website via HTTP will be automatically redirected to HTTPS via 301. + +It works with both GitLab's default domain and with your custom +domain (as long as you've set a valid certificate for it). + +To enable this setting: + +1. Navigate to your project's **Settings > Pages**. +1. Tick the checkbox **Force HTTPS (requires valid certificates)**. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md new file mode 100644 index 00000000000..255d535645d --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -0,0 +1,87 @@ +--- +type: reference +description: "Automatic Let's Encrypt SSL certificates for GitLab Pages." +--- + +# GitLab Pages integration with Let's Encrypt + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28996) in GitLab 12.1. + +The GitLab Pages integration with Let's Encrypt (LE) allows you +to use LE certificates for your Pages website with custom domains +without the hassle of having to issue and update them yourself; +GitLab does it for you, out-of-the-box. + +[Let's Encrypt](https://letsencrypt.org) is a free, automated, and +open source Certificate Authority. + +CAUTION: **Caution:** +This feature is in **beta** and might present bugs and UX issues +such as [#64870](https://gitlab.com/gitlab-org/gitlab-ce/issues/64870). +See all the related issues linked from this [issue's description](https://gitlab.com/gitlab-org/gitlab-ce/issues/28996) +for more information. + +## Requirements + +Before you can enable automatic provisioning of a SSL certificate for your domain, make sure you have: + +- Created a [project](../getting_started_part_two.md) in GitLab + containing your website's source code. +- Acquired a domain (`example.com`) and added a [DNS entry](index.md) + pointing it to your Pages website. +- [Added your domain to your Pages project](index.md#1-add-a-custom-domain-to-pages) + and verified your ownership. +- Have your website up and running, accessible through your custom domain. + +NOTE: **Note:** +GitLab's Let's Encrypt integration is enabled and available on GitLab.com. +For **self-managed** GitLab instances, make sure your administrator has +[enabled it](../../../../administration/pages/index.md#lets-encrypt-integration). + +## Enabling Let's Encrypt integration for your custom domain + +Once you've met the requirements, to enable Let's Encrypt integration: + +1. Navigate to your project's **Settings > Pages**. +1. Find your domain and click **Details**. +1. Click **Edit** in the top-right corner. +1. Enable Let's Encrypt integration by switching **Automatic certificate management using Let's Encrypt**: + +  + +1. Click **Save changes**. + +Once enabled, GitLab will obtain a LE certificate and add it to the +associated Pages domain. It will be also renewed automatically by GitLab. + +> **Notes:** +> +> - Issuing the certificate and updating Pages configuration +> **can take up to an hour**. +> - If you already have SSL certificate in domain settings it +> will continue to work until it will be replaced by Let's Encrypt's certificate. + +## Troubleshooting + +### Error "Certificate misses intermediates" + +If you get an error **Certificate misses intermediates** while trying to enable Let's Encrypt integration for your domain, follow the steps below: + +1. Go to your project's **Settings > Pages**. +1. Turn off **Force HTTPS** if it's turned on. +1. Click **Details** on your domain. +1. Click the **Edit** button in the top right corner of domain details page. +1. Enable Let's Encrypt integration. +1. Click **Save**. +1. Go to your project's **Settings > Pages**. +1. Turn on **Force HTTPS**. + +<!-- Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md new file mode 100644 index 00000000000..ee0550bfca2 --- /dev/null +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md @@ -0,0 +1,75 @@ +--- +type: concepts +--- + +# SSL/TLS Certificates + +_Read this document for a brief overview of SSL/TLS certificates in +the scope of GitLab Pages, for beginners in web development._ + +Every GitLab Pages project on GitLab.com will be available under +HTTPS for the default Pages domain (`*.gitlab.io`). Once you set +up your Pages project with your custom (sub)domain, if you want +it secured by HTTPS, you will have to issue a certificate for that +(sub)domain and install it on your project. + +NOTE: **Note:** +Certificates are NOT required to add to your custom +(sub)domain on your GitLab Pages project, though they are +highly recommendable. + +Let's start with an introduction to the importance of HTTPS. + +## Why should I care about HTTPS? + +This might be your first question. If our sites are hosted by GitLab Pages, +they are static, hence we are not dealing with server-side scripts +nor credit card transactions, then why do we need secure connections? + +Back in the 1990s, where HTTPS came out, [SSL](https://en.wikipedia.org/wiki/Transport_Layer_Security#SSL_1.0.2C_2.0_and_3.0) was considered a "special" +security measure, necessary just for big companies, like banks and shoppings sites +with financial transactions. +Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [ISRG](https://en.wikipedia.org/wiki/Internet_Security_Research_Group): + +> _We’ve since come to realize that HTTPS is important for almost all websites. It’s important for any website that allows people to log in with a password, any website that [tracks its users](https://www.washingtonpost.com/news/the-switch/wp/2013/12/10/nsa-uses-google-cookies-to-pinpoint-targets-for-hacking/) in any way, any website that [doesn’t want its content altered](http://arstechnica.com/tech-policy/2014/09/why-comcasts-javascript-ad-injections-threaten-security-net-neutrality/), and for any site that offers content people might not want others to know they are consuming. We’ve also learned that any site not secured by HTTPS [can be used to attack other sites](https://krebsonsecurity.com/2015/04/dont-be-fodder-for-chinas-great-cannon/)._ + +Therefore, the reason why certificates are so important is that they encrypt +the connection between the **client** (you, me, your visitors) +and the **server** (where you site lives), through a keychain of +authentications and validations. + +How about taking Josh's advice and protecting our sites too? We will be +well supported, and we'll contribute to a safer internet. + +## Organizations supporting HTTPS + +There is a huge movement in favor of securing all the web. W3C fully +[supports the cause](https://w3ctag.github.io/web-https/) and explains very well +the reasons for that. Richard Barnes, a writer for Mozilla Security Blog, +suggested that [Firefox would deprecate HTTP](https://blog.mozilla.org/security/2015/04/30/deprecating-non-secure-http/), +and would no longer accept unsecured connections. Recently, Mozilla published a +[communication](https://blog.mozilla.org/security/2016/03/29/march-2016-ca-communication/) +reiterating the importance of HTTPS. + +## Issuing Certificates + +GitLab Pages accepts certificates provided in the [PEM](https://support.quovadisglobal.com/kb/a37/what-is-pem-format.aspx) format, issued by +[Certificate Authorities (CAs)](https://en.wikipedia.org/wiki/Certificate_authority) or as +[self-signed certificates](https://en.wikipedia.org/wiki/Self-signed_certificate). Note that [self-signed certificates are typically not used](https://securingtomorrow.mcafee.com/other-blogs/mcafee-labs/self-signed-certificates-secure-so-why-ban/) +for public websites for security reasons and to ensure that browsers trust your site's certificate. + +There are various kinds of certificates, each one +with a certain security level. A static personal website will +not require the same security level as an online banking web app, +for instance. + +There are some certificate authorities that +offer free certificates, aiming to make the internet more secure +to everyone. The most popular is [Let's Encrypt](https://letsencrypt.org/), +which issues certificates trusted by most of browsers, it's open +source, and free to use. See [GitLab Pages integration with Let's Encrypt](../custom_domains_ssl_tls_certification/lets_encrypt_integration.md) to enable HTTPS on your custom domain. + +Similarly popular are [certificates issued by CloudFlare](https://www.cloudflare.com/ssl/), +which also offers a [free CDN service](https://blog.cloudflare.com/cloudflares-free-cdn-and-you/). +Their certs are valid up to 15 years. See the tutorial on +[how to add a CloudFlare Certificate to your GitLab Pages website](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/). diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index bc9a11504cd..9bc9fe97fd3 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1,314 +1 @@ ---- -last_updated: 2019-06-25 -type: concepts, reference, howto ---- - -# GitLab Pages custom domains and SSL/TLS Certificates - -Setting up GitLab Pages with custom domains, and adding SSL/TLS certificates to them, are optional features of GitLab Pages. - -These steps assume you've already [set your site up](getting_started_part_two.md) and it's served under the default Pages domain `namespace.gitlab.io`, or `namespace.gitlab.io/project-name`. - -## Adding your custom domain to GitLab Pages - -To use one or more custom domain with your Pages site, there are two things -you should consider first, which we'll cover in this guide: - -1. Either if you're adding a **root domain** or a **subdomain**, for which - you'll need to set up [DNS records](#dns-records) -1. Whether you want to add an [SSL/TLS certificate](#ssltls-certificates) or not - -To finish the association, you need to [add your domain to your project's Pages settings](#add-your-custom-domain-to-gitlab-pages-settings). - -Let's start from the beginning with [DNS records](#dns-records). -If you already know how they work and want to skip the introduction to DNS, -you may be interested in skipping it until the [TL;DR](#tldr) section below. - -### DNS Records - -A Domain Name System (DNS) web service routes visitors to websites -by translating domain names (such as `www.example.com`) into the -numeric IP addresses (such as `192.0.2.1`) that computers use to -connect to each other. - -A DNS record is created to point a (sub)domain to a certain location, -which can be an IP address or another domain. In case you want to use -GitLab Pages with your own (sub)domain, you need to access your domain's -registrar control panel to add a DNS record pointing it back to your -GitLab Pages site. - -Note that **how to** add DNS records depends on which server your domain -is hosted on. Every control panel has its own place to do it. If you are -not an admin of your domain, and don't have access to your registrar, -you'll need to ask for the technical support of your hosting service -to do it for you. - -To help you out, we've gathered some instructions on how to do that -for the most popular hosting services: - -- [Amazon](http://docs.aws.amazon.com/gettingstarted/latest/swh/getting-started-configure-route53.html) -- [Bluehost](https://my.bluehost.com/cgi/help/559) -- [CloudFlare](https://support.cloudflare.com/hc/en-us/articles/200169096-How-do-I-add-A-records-) -- [cPanel](https://documentation.cpanel.net/display/ALD/Edit+DNS+Zone) -- [DreamHost](https://help.dreamhost.com/hc/en-us/articles/215414867-How-do-I-add-custom-DNS-records-) -- [Go Daddy](https://www.godaddy.com/help/add-an-a-record-19238) -- [Hostgator](http://support.hostgator.com/articles/changing-dns-records) -- [Inmotion hosting](https://my.bluehost.com/cgi/help/559) -- [Media Temple](https://mediatemple.net/community/products/dv/204403794/how-can-i-change-the-dns-records-for-my-domain) -- [Microsoft](https://msdn.microsoft.com/en-us/library/bb727018.aspx) - -If your hosting service is not listed above, you can just try to -search the web for `how to add dns record on <my hosting service>`. - -#### DNS A record - -In case you want to point a root domain (`example.com`) to your -GitLab Pages site, deployed to `namespace.gitlab.io`, you need to -log into your domain's admin control panel and add a DNS `A` record -pointing your domain to Pages' server IP address. For projects on -GitLab.com, this IP is `35.185.44.232`. For projects living in -other GitLab instances (CE or EE), please contact your sysadmin -asking for this information (which IP address is Pages server -running on your instance). - -**Practical Example:** - - - -CAUTION: **Caution:** -Note that if you use your root domain for your GitLab Pages website -**only**, and if your domain registrar supports this feature, you can -add a DNS apex `CNAME` record instead of an `A` record. The main -advantage of doing so is that when GitLab Pages IP on GitLab.com -changes for whatever reason, you don't need to update your `A` record. -There may be a few exceptions, but **this method is not recommended** -as it most likely won't work if you set an `MX` record for your root domain. - -#### DNS CNAME record - -In case you want to point a subdomain (`hello-world.example.com`) -to your GitLab Pages site initially deployed to `namespace.gitlab.io`, -you need to log into your domain's admin control panel and add a DNS -`CNAME` record pointing your subdomain to your website URL -(`namespace.gitlab.io`) address. - -Note that, whether it's a user or a project website, the `CNAME` -should point to your Pages domain (`namespace.gitlab.io`), -without any `/project-name`. - -**Practical Example:** - - - -#### DNS TXT record - -Unless your GitLab administrator has [disabled custom domain verification](../../../administration/pages/index.md#custom-domain-verification), -you'll have to prove that you own the domain by creating a `TXT` record -containing a verification code. The code will be displayed after you -[add your custom domain to GitLab Pages settings](#add-your-custom-domain-to-gitlab-pages-settings). - -If using a [DNS A record](#dns-a-record), you can place the TXT record directly -under the domain. If using a [DNS CNAME record](#dns-cname-record), the two record types won't -co-exist, so you need to place the TXT record in a special subdomain of its own. - -If the domain cannot be verified for 7 days, it will be removed from the GitLab project. - -#### TL;DR - -For root domains (`domain.com`), set a DNS `A` record and verify your -domain's ownership with a TXT record: - -| From | DNS Record | To | -| ---- | ---------- | -- | -| domain.com | A | 35.185.44.232 | -| domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | - -For subdomains (`subdomain.domain.com`), set a DNS `CNAME` record and -verify your domain's ownership with a TXT record: - -| From | DNS Record | To | -| ---- | ---------- | -- | -| subdomain.domain.com | CNAME | namespace.gitlab.io | -| _gitlab-pages-verification-code.subdomain.domain.com | TXT | gitlab-pages-verification-code=00112233445566778899aabbccddeeff | - -> **Notes**: -> -> - **Do not** use a CNAME record if you want to point your - `domain.com` to your GitLab Pages site. Use an `A` record instead. -> - **Do not** add any special chars after the default Pages - domain. E.g., **do not** point your `subdomain.domain.com` to - `namespace.gitlab.io.` or `namespace.gitlab.io/`. -> - GitLab Pages IP on GitLab.com [was changed](https://about.gitlab.com/2017/03/06/we-are-changing-the-ip-of-gitlab-pages-on-gitlab-com/) in 2017. -> - GitLab Pages IP on GitLab.com [has been changed](https://about.gitlab.com/2018/07/19/gcp-move-update/#gitlab-pages-and-custom-domains) - from `52.167.214.135` to `35.185.44.232` in 2018. - -### Add your custom domain to GitLab Pages settings - -Once you've set the DNS record, you'll need navigate to your project's -**Setting > Pages** and click **+ New domain** to add your custom domain to -GitLab Pages. You can choose whether to add an [SSL/TLS certificate](#ssltls-certificates) -to make your website accessible under HTTPS or leave it blank. If you don't add a certificate, -your site will be accessible only via HTTP: - - - -Once you have added a new domain, you will need to **verify your ownership** -(unless the GitLab administrator has disabled this feature). A verification code -will be shown to you; add it as a [DNS TXT record](#dns-txt-record), then press -the "Verify ownership" button to activate your new domain: - - - -Once your domain has been verified, leave the verification record in place - -your domain will be periodically reverified, and may be disabled if the record -is removed. - -You can add more than one alias (custom domains and subdomains) to the same project. -An alias can be understood as having many doors leading to the same room. - -All the aliases you've set to your site will be listed on **Setting > Pages**. -From that page, you can view, add, and remove them. - -Note that [DNS propagation may take some time (up to 24h)](http://www.inmotionhosting.com/support/domain-names/dns-nameserver-changes/domain-names-dns-changes), -although it's usually a matter of minutes to complete. Until it does, verification -will fail and attempts to visit your domain will respond with a 404. - -### Redirecting `www.domain.com` to `domain.com` with Cloudflare - -If you use Cloudflare, you can redirect `www` to `domain.com` without the need of adding both -`www.domain.com` and `domain.com` to GitLab. This happens due to a [Cloudflare feature that creates -a 301 redirect as a "page rule"](https://gitlab.com/gitlab-org/gitlab-ce/issues/48848#note_87314849) for redirecting `www.domain.com` to `domain.com`. In this case, -you can use the following setup: - -- In Cloudflare, create a DNS `A` record pointing `domain.com` to `35.185.44.232` -- In GitLab, add the domain to GitLab Pages -- In Cloudflare, create a DNS `TXT` record to verify your domain -- In Cloudflare, create a DNS `CNAME` record pointing `www` to `domain.com` - -## SSL/TLS Certificates - -Every GitLab Pages project on GitLab.com will be available under -HTTPS for the default Pages domain (`*.gitlab.io`). Once you set -up your Pages project with your custom (sub)domain, if you want -it secured by HTTPS, you will have to issue a certificate for that -(sub)domain and install it on your project. - ->**Note:** -Certificates are NOT required to add to your custom -(sub)domain on your GitLab Pages project, though they are -highly recommendable. - -Let's start with an introduction to the importance of HTTPS. -Alternatively, jump ahead to [adding certificates to your project](#adding-certificates-to-pages). - -### Why should I care about HTTPS? - -This might be your first question. If our sites are hosted by GitLab Pages, -they are static, hence we are not dealing with server-side scripts -nor credit card transactions, then why do we need secure connections? - -Back in the 1990s, where HTTPS came out, [SSL](https://en.wikipedia.org/wiki/Transport_Layer_Security#SSL_1.0.2C_2.0_and_3.0) was considered a "special" -security measure, necessary just for big companies, like banks and shoppings sites -with financial transactions. -Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [ISRG](https://en.wikipedia.org/wiki/Internet_Security_Research_Group): - -> _We’ve since come to realize that HTTPS is important for almost all websites. It’s important for any website that allows people to log in with a password, any website that [tracks its users](https://www.washingtonpost.com/news/the-switch/wp/2013/12/10/nsa-uses-google-cookies-to-pinpoint-targets-for-hacking/) in any way, any website that [doesn’t want its content altered](http://arstechnica.com/tech-policy/2014/09/why-comcasts-javascript-ad-injections-threaten-security-net-neutrality/), and for any site that offers content people might not want others to know they are consuming. We’ve also learned that any site not secured by HTTPS [can be used to attack other sites](https://krebsonsecurity.com/2015/04/dont-be-fodder-for-chinas-great-cannon/)._ - -Therefore, the reason why certificates are so important is that they encrypt -the connection between the **client** (you, me, your visitors) -and the **server** (where you site lives), through a keychain of -authentications and validations. - -How about taking Josh's advice and protecting our sites too? We will be -well supported, and we'll contribute to a safer internet. - -### Organizations supporting HTTPS - -There is a huge movement in favor of securing all the web. W3C fully -[supports the cause](https://w3ctag.github.io/web-https/) and explains very well -the reasons for that. Richard Barnes, a writer for Mozilla Security Blog, -suggested that [Firefox would deprecate HTTP](https://blog.mozilla.org/security/2015/04/30/deprecating-non-secure-http/), -and would no longer accept unsecured connections. Recently, Mozilla published a -[communication](https://blog.mozilla.org/security/2016/03/29/march-2016-ca-communication/) -reiterating the importance of HTTPS. - -## Issuing Certificates - -GitLab Pages accepts certificates provided in the [PEM](https://support.quovadisglobal.com/kb/a37/what-is-pem-format.aspx) format, issued by -[Certificate Authorities (CAs)](https://en.wikipedia.org/wiki/Certificate_authority) or as -[self-signed certificates](https://en.wikipedia.org/wiki/Self-signed_certificate). Note that [self-signed certificates are typically not used](https://securingtomorrow.mcafee.com/other-blogs/mcafee-labs/self-signed-certificates-secure-so-why-ban/) -for public websites for security reasons and to ensure that browsers trust your site's certificate. - -There are various kinds of certificates, each one -with a certain security level. A static personal website will -not require the same security level as an online banking web app, -for instance. - -There are some certificate authorities that -offer free certificates, aiming to make the internet more secure -to everyone. The most popular is [Let's Encrypt](https://letsencrypt.org/), -which issues certificates trusted by most of browsers, it's open -source, and free to use. See our tutorial on [how to secure your GitLab Pages website with Let's Encrypt](lets_encrypt_for_gitlab_pages.md). - -Similarly popular are [certificates issued by CloudFlare](https://www.cloudflare.com/ssl/), -which also offers a [free CDN service](https://blog.cloudflare.com/cloudflares-free-cdn-and-you/). -Their certs are valid up to 15 years. See the tutorial on -[how to add a CloudFlare Certificate to your GitLab Pages website](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/). - -### Adding certificates to Pages - -Regardless the CA you choose, the steps to add your certificate to -your Pages project are the same. - -#### Requirements - -1. A PEM certificate -1. An intermediate certificate -1. A private key - - - -These fields are found under your **Project**'s **Settings** > **Pages** > **New Domain**. - -#### Certificate types - -- A PEM certificate is the certificate generated by the CA, - which needs to be added to the field **Certificate (PEM)**. -- An [intermediate certificate](https://en.wikipedia.org/wiki/Intermediate_certificate_authority) (aka "root certificate") is - the part of the encryption keychain that identifies the CA. - Usually it's combined with the PEM certificate, but there are - some cases in which you need to add them manually. - [CloudFlare certs](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) - are one of these cases. -- A private key is an encrypted key which validates - your PEM against your domain. - -#### Add the certificate to your project - -Once you've met the requirements: - -- Your PEM certificate needs to be added to the first field. -- If your certificate is missing its intermediate, copy - and paste the root certificate (usually available from your CA website) - and paste it in the [same field as your PEM certificate](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/), - just jumping a line between them. -- Copy your private key and paste it in the last field. - -NOTE: **Note:** -**Do not** open certificates or encryption keys in -regular text editors. Always use code editors (such as -Sublime Text, Atom, Dreamweaver, Brackets, etc). - -## Force HTTPS for GitLab Pages websites - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/28857) in GitLab 10.7. - -To make your website's visitors even more secure, you can choose to -force HTTPS for GitLab Pages. By doing so, all attempts to visit your -website via HTTP will be automatically redirected to HTTPS via 301. - -It works with both GitLab's default domain and with your custom -domain (as long as you've set a valid certificate for it). - -To enable this setting, navigate to your project's **Settings > Pages** -and tick the checkbox **Force HTTPS (requires valid certificates)**. +This document was moved to [another location](custom_domains_ssl_tls_certification/index.md). diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index fe92d19567d..3bab652ca39 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -22,7 +22,7 @@ Optional Features: 1. **Optional**: an SSL/TLS certificate so your custom domain is accessible under HTTPS. -The optional settings, custom domain, DNS records, and SSL/TLS certificates, are described in [Part 3](getting_started_part_three.md)). +The optional settings, custom domain, DNS records, and SSL/TLS certificates, are described in [GitLab Pages custom domains and SSL/TLS Certificates](custom_domains_ssl_tls_certification/index.md)). ## Project @@ -86,17 +86,17 @@ You can also take some **optional** further steps: - _Remove the fork relationship._ The fork relationship is necessary to contribute back to the project you originally forked from. If you don't have any intentions to do so, you can remove it. To do so, navigate to your project's **Settings**, expand **Advanced settings**, and scroll down to **Remove fork relationship**: -  +  - _Make it a user or group website._ To turn a **project website** forked from the Pages group into a **user/group** website, you'll need to: - - Rename it to `namespace.gitlab.io`: go to your project's - **Settings > General** and expand **Advanced**. Scroll down to - **Rename repository** and change the path to `namespace.gitlab.io`. - - Adjust your SSG's [base URL](#urls-and-baseurls) from `"project-name"` to - `""`. This setting will be at a different place for each SSG, as each of them - have their own structure and file tree. Most likely, it will be in the SSG's - config file. + - Rename it to `namespace.gitlab.io`: go to your project's + **Settings > General** and expand **Advanced**. Scroll down to + **Rename repository** and change the path to `namespace.gitlab.io`. + - Adjust your SSG's [base URL](#urls-and-baseurls) from `"project-name"` to + `""`. This setting will be at a different place for each SSG, as each of them + have their own structure and file tree. Most likely, it will be in the SSG's + config file. ### Create a project from scratch @@ -107,12 +107,12 @@ You can also take some **optional** further steps: files to your project, add, commit and push to GitLab. 1. From the your **Project**'s page, click **Set up CI/CD**: -  +  1. Choose one of the templates from the dropbox menu. Pick up the template corresponding to the SSG you're using (or plain HTML). -  +  Once you have both site files and `.gitlab-ci.yml` in your project's root, GitLab CI/CD will build your site and deploy it with Pages. @@ -123,20 +123,20 @@ where you'll find its default URL. > **Notes:** > > - GitLab Pages [supports any SSG](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/), but, - if you don't find yours among the templates, you'll need - to configure your own `.gitlab-ci.yml`. To do that, please - read through the article [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md). New SSGs are very welcome among - the [example projects](https://gitlab.com/pages). If you set - up a new one, please - [contribute](https://gitlab.com/pages/pages.gitlab.io/blob/master/CONTRIBUTING.md) - to our examples. +> if you don't find yours among the templates, you'll need +> to configure your own `.gitlab-ci.yml`. To do that, please +> read through the article [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md). New SSGs are very welcome among +> the [example projects](https://gitlab.com/pages). If you set +> up a new one, please +> [contribute](https://gitlab.com/pages/pages.gitlab.io/blob/master/CONTRIBUTING.md) +> to our examples. > > - The second step _"Clone it to your local computer"_, can be done - differently, achieving the same results: instead of cloning the bare - repository to you local computer and moving your site files into it, - you can run `git init` in your local website directory, add the - remote URL: `git remote add origin git@gitlab.com:namespace/project-name.git`, - then add, commit, and push to GitLab. +> differently, achieving the same results: instead of cloning the bare +> repository to you local computer and moving your site files into it, +> you can run `git init` in your local website directory, add the +> remote URL: `git remote add origin git@gitlab.com:namespace/project-name.git`, +> then add, commit, and push to GitLab. ## URLs and Baseurls @@ -169,4 +169,4 @@ baseurl: "" ## Custom Domains GitLab Pages supports custom domains and subdomains, served under HTTP or HTTPS. -See [GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md) for more information. +See [GitLab Pages custom domains and SSL/TLS Certificates](custom_domains_ssl_tls_certification/index.md) for more information. diff --git a/doc/user/project/pages/img/pages_project_templates_11-8.png b/doc/user/project/pages/img/pages_project_templates_11-8.png Binary files differdeleted file mode 100644 index a645d28260b..00000000000 --- a/doc/user/project/pages/img/pages_project_templates_11-8.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_project_templates_v11_8.png b/doc/user/project/pages/img/pages_project_templates_v11_8.png Binary files differnew file mode 100644 index 00000000000..61cae88b5a8 --- /dev/null +++ b/doc/user/project/pages/img/pages_project_templates_v11_8.png diff --git a/doc/user/project/pages/img/remove_pages.png b/doc/user/project/pages/img/remove_pages.png Binary files differindex 60f76f15f93..d6c37ef30cd 100644 --- a/doc/user/project/pages/img/remove_pages.png +++ b/doc/user/project/pages/img/remove_pages.png diff --git a/doc/user/project/pages/img/verify_your_domain.png b/doc/user/project/pages/img/verify_your_domain.png Binary files differdeleted file mode 100644 index d870f9e6505..00000000000 --- a/doc/user/project/pages/img/verify_your_domain.png +++ /dev/null diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 64b1e259292..a0bc10b0ed7 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -101,7 +101,7 @@ To get started with GitLab Pages, you can either: 1. Select **Create from Template**. 1. Choose one of the templates starting with **Pages**: -  +  1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your @@ -115,8 +115,8 @@ will run a new pipeline to publish your changes to the server. _Advanced options:_ -- [Use a custom domain](getting_started_part_three.md#adding-your-custom-domain-to-gitlab-pages) -- Apply [SSL/TLS certification](getting_started_part_three.md#ssltls-certificates) to your custom domain +- [Use a custom domain](custom_domains_ssl_tls_certification/index.md#set-up-pages-with-a-custom-domain) +- Apply [SSL/TLS certification](custom_domains_ssl_tls_certification/index.md#adding-an-ssltls-certificate-to-pages) to your custom domain ## Availability @@ -142,9 +142,9 @@ To learn more about configuration options for GitLab Pages, read the following: | [GitLab CI/CD for GitLab Pages](getting_started_part_four.md) | Understand how to create your own `.gitlab-ci.yml` for your site. | | [Exploring GitLab Pages](introduction.md) | Requirements, technical aspects, specific GitLab CI's configuration options, Access Control, custom 404 pages, limitations, FAQ. | |---+---| -| [Custom domains and SSL/TLS Certificates](getting_started_part_three.md) | How to add custom domains and subdomains to your website, configure DNS records and SSL/TLS certificates. | +| [Custom domains and SSL/TLS Certificates](custom_domains_ssl_tls_certification/index.md) | How to add custom domains and subdomains to your website, configure DNS records and SSL/TLS certificates. | +| [Let's Encrypt integration](custom_domains_ssl_tls_certification/lets_encrypt_integration.md) | Secure your Pages sites with Let's Encrypt certificates automatically obtained and renewed by GitLab. | | [CloudFlare certificates](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) | Secure your Pages site with CloudFlare certificates. | -| [Let's Encrypt certificates](lets_encrypt_for_gitlab_pages.md) | Secure your Pages site with Let's Encrypt certificates. | |---+---| | [Static vs dynamic websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) | A conceptual overview on static versus dynamic sites. | | [Modern static site generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) | A conceptual overview on SSGs. | diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 9451b5349c0..e8984cb8b9f 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -83,23 +83,23 @@ You can enable Pages access control on your project, so that only 1. Navigate to your project's **Settings > General > Permissions**. 1. Toggle the **Pages** button to enable the access control. - NOTE: **Note:** - If you don't see the toggle button, that means that it's not enabled. - Ask your administrator to [enable it](../../../administration/pages/index.md#access-control). + NOTE: **Note:** + If you don't see the toggle button, that means that it's not enabled. + Ask your administrator to [enable it](../../../administration/pages/index.md#access-control). 1. The Pages access control dropdown allows you to set who can view pages hosted with GitLab Pages, depending on your project's visibility: - - If your project is private: - - **Only project members**: Only project members will be able to browse the website. - - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. - - If your project is internal: - - **Only project members**: Only project members will be able to browse the website. - - **Everyone with access**: Everyone logged into GitLab will be able to browse the website, no matter their project membership. - - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. - - If your project is public: - - **Only project members**: Only project members will be able to browse the website. - - **Everyone with access**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. + - If your project is private: + - **Only project members**: Only project members will be able to browse the website. + - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. + - If your project is internal: + - **Only project members**: Only project members will be able to browse the website. + - **Everyone with access**: Everyone logged into GitLab will be able to browse the website, no matter their project membership. + - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. + - If your project is public: + - **Only project members**: Only project members will be able to browse the website. + - **Everyone with access**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. 1. Click **Save changes**. diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md index 91a660c0f7a..1338c7e58f5 100644 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md @@ -1,10 +1,15 @@ --- -description: "How to secure GitLab Pages websites with Let's Encrypt." +description: "How to secure GitLab Pages websites with Let's Encrypt (manual process, deprecated)." type: howto -last_updated: 2019-06-04 +last_updated: 2019-07-15 --- -# Let's Encrypt for GitLab Pages +# Let's Encrypt for GitLab Pages (manual process, deprecated) + +CAUTION: **Warning:** +This method is still valid but was **deprecated** in favor of the +[Let's Encrypt integration](custom_domains_ssl_tls_certification/lets_encrypt_integration.md) +introduced in GitLab 12.1. If you have a GitLab Pages website served under your own domain, you might want to secure it with a SSL/TSL certificate. @@ -18,9 +23,9 @@ To follow along with this tutorial, we assume you already have: - Created a [project](getting_started_part_two.md) in GitLab which contains your website's source code. -- Acquired a domain (`example.com`) and added a [DNS entry](getting_started_part_three.md#dns-records) +- Acquired a domain (`example.com`) and added a [DNS entry](custom_domains_ssl_tls_certification/index.md#set-up-pages-with-a-custom-domain) pointing it to your Pages website. -- [Added your domain to your Pages project](getting_started_part_three.md#add-your-custom-domain-to-gitlab-pages-settings) +- [Added your domain to your Pages project](custom_domains_ssl_tls_certification/index.md#steps) and verified your ownership. - Cloned your project into your computer. - Your website up and running, served under HTTP protocol at `http://example.com`. @@ -36,111 +41,111 @@ operating systems the steps might be slightly different. Follow the [CertBot instructions](https://certbot.eff.org/) according to your OS. 1. On your computer, open a terminal and navigate to your repository's - root directory: + root directory: - ```bash - cd path/to/dir - ``` + ```bash + cd path/to/dir + ``` 1. Install CertBot (the tool Let's Encrypt uses to issue certificates): - ```bash - brew install certbot - ``` + ```bash + brew install certbot + ``` 1. Request a certificate for your domain (`example.com`) and - provide an email account (`your@email.com`) to receive notifications: + provide an email account (`your@email.com`) to receive notifications: - ```bash - sudo certbot certonly -a manual -d example.com --email your@email.com - ``` + ```bash + sudo certbot certonly -a manual -d example.com --email your@email.com + ``` - Alternatively, you can register without adding an e-mail account, - but you won't be notified about the certificate expiration's date: + Alternatively, you can register without adding an e-mail account, + but you won't be notified about the certificate expiration's date: - ```bash - sudo certbot certonly -a manual -d example.com --register-unsafely-without-email - ``` + ```bash + sudo certbot certonly -a manual -d example.com --register-unsafely-without-email + ``` - TIP: **Tip:** - Read through CertBot's documentation on their - [command line options](https://certbot.eff.org/docs/using.html#certbot-command-line-options). + TIP: **Tip:** + Read through CertBot's documentation on their + [command line options](https://certbot.eff.org/docs/using.html#certbot-command-line-options). 1. You'll be prompted with a message to agree with their terms. - Press `A` to agree and `Y` to let they log your IP. + Press `A` to agree and `Y` to let they log your IP. - CertBot will then prompt you with the following message: + CertBot will then prompt you with the following message: - ```bash - Create a file containing just this data: + ```bash + Create a file containing just this data: - Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP.HUGNKk82jlsmOOfphlt8Jy69iuglsn095nxOMH9j3Yb + Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP.HUGNKk82jlsmOOfphlt8Jy69iuglsn095nxOMH9j3Yb - And make it available on your web server at this URL: + And make it available on your web server at this URL: - http://example.com/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP + http://example.com/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP - Press Enter to Continue - ``` + Press Enter to Continue + ``` 1. **Do not press Enter yet.** Let's Encrypt will need to verify your - domain ownership before issuing the certificate. To do so, create 3 - consecutive directories under your website's root: - `/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP/` - and add to the last folder an `index.html` file containing the content - referred on the previous prompt message: - - ```bash - Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP.HUGNKk82jlsmOOfphlt8Jy69iuglsn095nxOMH9j3Yb - ``` - - Note that this file needs to be accessed under - `http://example.com/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP` - to allow Let's Encrypt to verify the ownership of your domain, - therefore, it needs to be part of the website content under the - repo's [`public`](index.md#how-it-works) folder. + domain ownership before issuing the certificate. To do so, create 3 + consecutive directories under your website's root: + `/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP/` + and add to the last folder an `index.html` file containing the content + referred on the previous prompt message: + + ```bash + Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP.HUGNKk82jlsmOOfphlt8Jy69iuglsn095nxOMH9j3Yb + ``` + + Note that this file needs to be accessed under + `http://example.com/.well-known/acme-challenge/Rxnv6WKo95hsuLVX3osmT6LgmzsJKSaK9htlPToohOP` + to allow Let's Encrypt to verify the ownership of your domain, + therefore, it needs to be part of the website content under the + repo's [`public`](index.md#how-it-works) folder. 1. Add, commit, and push the file into your repo in GitLab. Once the pipeline - passes, press **Enter** on your terminal to continue issuing your - certificate. CertBot will then prompt you with the following message: - - ```bash - Waiting for verification... - Cleaning up challenges - - IMPORTANT NOTES: - - Congratulations! Your certificate and chain have been saved at: - /etc/letsencrypt/live/example.com/fullchain.pem - Your key file has been saved at: - /etc/letsencrypt/live/example.com/privkey.pem - Your cert will expire on 2019-03-12. To obtain a new or tweaked - version of this certificate in the future, simply run certbot - again. To non-interactively renew *all* of your certificates, run - "certbot renew" - - If you like Certbot, please consider supporting our work by: - - Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate - Donating to EFF: https://eff.org/donate-le - ``` + passes, press **Enter** on your terminal to continue issuing your + certificate. CertBot will then prompt you with the following message: + + ```bash + Waiting for verification... + Cleaning up challenges + + IMPORTANT NOTES: + - Congratulations! Your certificate and chain have been saved at: + /etc/letsencrypt/live/example.com/fullchain.pem + Your key file has been saved at: + /etc/letsencrypt/live/example.com/privkey.pem + Your cert will expire on 2019-03-12. To obtain a new or tweaked + version of this certificate in the future, simply run certbot + again. To non-interactively renew *all* of your certificates, run + "certbot renew" + - If you like Certbot, please consider supporting our work by: + + Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate + Donating to EFF: https://eff.org/donate-le + ``` ## Add your certificate to GitLab Pages Now that your certificate has been issued, let's add it to your Pages site: 1. Back at GitLab, navigate to your project's **Settings > Pages**, - find your domain and click **Details** and **Edit** to add your certificate. + find your domain and click **Details** and **Edit** to add your certificate. 1. From your terminal, copy and paste the certificate into the first field - **Certificate (PEM)**: + **Certificate (PEM)**: - ```bash - sudo cat /etc/letsencrypt/live/example.com/fullchain.pem | pbcopy - ``` + ```bash + sudo cat /etc/letsencrypt/live/example.com/fullchain.pem | pbcopy + ``` 1. Copy and paste the private key into the second field **Key (PEM)**: - ```bash - sudo cat /etc/letsencrypt/live/example.com/privkey.pem | pbcopy - ``` + ```bash + sudo cat /etc/letsencrypt/live/example.com/privkey.pem | pbcopy + ``` 1. Click **Save changes** to apply them to your website. 1. Wait a few minutes for the configuration changes to take effect. diff --git a/doc/user/project/pipelines/img/pipeline_schedule_variables.png b/doc/user/project/pipelines/img/pipeline_schedule_variables.png Binary files differindex 29846206491..ce3c3dc6af1 100644 --- a/doc/user/project/pipelines/img/pipeline_schedule_variables.png +++ b/doc/user/project/pipelines/img/pipeline_schedule_variables.png diff --git a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png Binary files differindex e135dd51070..993fbf8ca00 100644 --- a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png +++ b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png diff --git a/doc/user/project/pipelines/job_artifacts.md b/doc/user/project/pipelines/job_artifacts.md index 1966b136e9d..ef90899c512 100644 --- a/doc/user/project/pipelines/job_artifacts.md +++ b/doc/user/project/pipelines/job_artifacts.md @@ -1,24 +1,24 @@ +--- +type: reference, howto +--- + # Introduction to job artifacts -> **Notes:** -> -> - Since GitLab 8.2 and GitLab Runner 0.7.0, job artifacts that are created by -> GitLab Runner are uploaded to GitLab and are downloadable as a single archive -> (`tar.gz`) using the GitLab UI. -> - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format -> changed to `ZIP`, and it is now possible to browse its contents, with the added -> ability of downloading the files separately. -> - Starting with GitLab 8.17, builds are renamed to jobs. -> - The artifacts browser will be available only for new artifacts that are sent -> to GitLab using GitLab Runner version 1.0 and up. It will not be possible to -> browse old artifacts already uploaded to GitLab. ->- This is the user documentation. For the administration guide see -> [administration/job_artifacts](../../../administration/job_artifacts.md). - -Artifacts is a list of files and directories which are attached to a job -after it finishes. This feature is enabled by default in all +> - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. +> - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`, and it is now possible to browse its contents, with the added ability of downloading the files separately. +> - In GitLab 8.17, builds were renamed to jobs. +> - The artifacts browser will be available only for new artifacts that are sent to GitLab using GitLab Runner version 1.0 and up. It will not be possible to browse old artifacts already uploaded to GitLab. + +Artifacts are a list of files and directories which created by a job +once it finishes. This feature is [enabled by default](../../../administration/job_artifacts.md) in all GitLab installations. +Job artifacts that are created by GitLab Runner are uploaded to GitLab and are downloadable as a single archive using the GitLab UI. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). +Watch also [GitLab CI pipeline tutorial for beginners](https://www.youtube.com/watch?v=Jav4vbUrqII). + ## Defining artifacts in `.gitlab-ci.yml` A simple example of using the artifacts definition in `.gitlab-ci.yml` would be @@ -50,11 +50,8 @@ For more examples on artifacts, follow the [artifacts reference in ## Browsing artifacts -> **Note:** > With GitLab 9.2, PDFs, images, videos and other formats can be previewed > directly in the job artifacts browser without the need to download them. -> -> **Note:** > With [GitLab 10.1][ce-14399], HTML files in a public project can be previewed > directly in a new tab without the need to download them when > [GitLab Pages](../../../administration/pages/index.md) is enabled. @@ -67,8 +64,6 @@ artifacts in case you changed your mind and want to keep them.  ---- - The archive browser shows the name and the actual file size of each file in the archive. If your artifacts contained directories, then you are also able to browse inside them. @@ -80,8 +75,6 @@ one HTML file that you can view directly online when  ---- - ## Downloading artifacts If you need to download the whole archive, there are buttons in various places @@ -90,29 +83,29 @@ inside GitLab that make that possible. 1. While on the pipelines page, you can see the download icon for each job's artifacts archive in the right corner: -  +  1. While on the **Jobs** page, you can see the download icon for each job's artifacts archive in the right corner: -  +  1. While inside a specific job, you are presented with a download button along with the one that browses the archive: -  +  1. And finally, when browsing an archive you can see the download button at the top right corner: -  +  ## Downloading the latest artifacts It is possible to download the latest artifacts of a job via a well known URL so you can use it for scripting purposes. ->**Note:** +NOTE: **Note:** The latest artifacts are considered as the artifacts created by jobs in the latest pipeline that succeeded for the specific ref. Artifacts for other pipelines can be accessed with direct access to them. @@ -173,9 +166,9 @@ https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/artifacts/master/file/htmlcov/ind The latest builds are also exposed in the UI in various places. Specifically, look for the download button in: -- the main project's page -- the branches page -- the tags page +- The main project's page +- The branches page +- The tags page If the latest job has failed to upload the artifacts, you can see that information in the UI. @@ -205,3 +198,15 @@ In order to retrieve a job artifact of a different project, you might need to us [expiry date]: ../../../ci/yaml/README.md#artifactsexpire_in [ce-14399]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14399 + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index dd5427ab35a..f3e9c950efd 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -1,10 +1,14 @@ +--- +type: reference, howto +--- + # Pipeline schedules -> **Notes**: -> > - Introduced in GitLab 9.1 as [Trigger Schedule](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10533). > - [Renamed to Pipeline Schedule](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10853) in GitLab 9.2. -> - Cron notation is parsed by [Fugit](https://github.com/floraison/fugit). + +NOTE: **Note:** +Cron notation is parsed by [Fugit](https://github.com/floraison/fugit). Pipelines are normally run based on certain conditions being met. For example, when a branch is pushed to repository. @@ -118,3 +122,15 @@ on the target branch, the schedule will stop creating new pipelines. This can happen if, for example, the owner is blocked or removed from the project, or the target branch or tag is protected. In this case, someone with sufficient privileges must take ownership of the schedule. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 24e15a37a40..456209c2180 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Pipelines settings To reach the pipelines settings navigate to your project's @@ -5,6 +9,10 @@ To reach the pipelines settings navigate to your project's The following settings can be configured per project. +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). +Watch also [GitLab CI pipeline tutorial for beginners](https://www.youtube.com/watch?v=Jav4vbUrqII). + ## Git strategy With Git strategy, you can choose the default way your repository is fetched @@ -25,10 +33,10 @@ in `.gitlab-ci.yml`. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28919) in GitLab 12.0. NOTE: **Note**: -As of GitLab 12.0, newly created projects will automatically have a default +As of GitLab 12.0, newly created projects will automatically have a default `git depth` value of `50`. -It is possible to limit the number of changes that GitLab CI/CD will fetch when cloning +It is possible to limit the number of changes that GitLab CI/CD will fetch when cloning a repository. Setting a limit to `git depth` can speed up Pipelines execution. Maximum allowed value is `1000`. @@ -68,6 +76,13 @@ Here are some valid examples: - `my/path/.gitlab-ci.yml` - `my/path/.my-custom-file.yml` +The path can be customized at a project level. To customize the path: + +1. Go to the project's **Settings > CI / CD**. +1. Expand the **General pipelines** section. +1. Provide a value in the **Custom CI config path** field. +1. Click **Save changes**. + ## Test coverage parsing If you use test coverage in your code, GitLab can capture its output in the @@ -89,6 +104,22 @@ in the jobs table. A few examples of known coverage tools for a variety of languages can be found in the pipelines settings page. +### Removing color codes + +Some test coverage tools output with ANSI color codes that won't be +parsed correctly by the regular expression and will cause coverage +parsing to fail. + +If your coverage tool doesn't provide an option to disable color +codes in the output, you can pipe the output of the coverage tool through a +small one line script that will strip the color codes off. + +For example: + +```bash +lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g' +``` + ## Visibility of pipelines Access to pipelines and job details (including output of logs and artifacts) @@ -200,3 +231,15 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st ## Environment Variables [Environment variables](../../../ci/variables/README.html#gitlab-cicd-environment-variables) can be set in an environment to be available to a runner. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 20a03dff2da..7a79cdbcaee 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Protected Branches [Permissions](../permissions.md) in GitLab are fundamentally defined around the @@ -9,13 +13,13 @@ created protected branches. By default, a protected branch does four simple things: -- it prevents its creation, if not already created, from everybody except users - with Maintainer permission -- it prevents pushes from everybody except users with Maintainer permission -- it prevents **anyone** from force pushing to the branch -- it prevents **anyone** from deleting the branch +- It prevents its creation, if not already created, from everybody except users + with Maintainer permission. +- It prevents pushes from everybody except users with Maintainer permission. +- It prevents **anyone** from force pushing to the branch. +- It prevents **anyone** from deleting the branch. ->**Note**: +NOTE: **Note:** A GitLab admin is allowed to push to the protected branches. See the [Changelog](#changelog) section for changes over time. @@ -30,11 +34,11 @@ that the `master` branch is protected by default. 1. From the **Branch** dropdown menu, select the branch you want to protect and click **Protect**. In the screenshot below, we chose the `develop` branch. -  +  1. Once done, the protected branch will appear in the "Protected branches" list. -  +  ## Using the Allowed to merge and Allowed to push settings @@ -68,7 +72,7 @@ they are set to "Maintainers" by default. ## Restricting push and merge access to certain users **(STARTER)** -> This feature was [introduced][ce-5081] in [GitLab Starter][ee] 8.11. +> [Introduced][ce-5081] in [GitLab Starter][ee] 8.11. With GitLab Enterprise Edition you can restrict access to protected branches by choosing a role (Maintainers, Developers) as well as certain users. From the @@ -141,7 +145,7 @@ branches via GitLab's web interface: 1. In order to prevent accidental deletion, an additional confirmation is required -  +  Deleting a protected branch is only allowed via the web interface, not via Git. This means that you can't accidentally delete a protected branch from your @@ -174,12 +178,21 @@ for details about the pipelines security model. - Allow developers to merge into a protected branch without having push access [gitlab-org/gitlab-ce!4892][ce-4892] - Allow specifying protected branches using wildcards [gitlab-org/gitlab-ce!4665][ce-4665] ---- - [ce-4665]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4665 "Allow specifying protected branches using wildcards" [ce-4892]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4892 "Allow developers to merge into a protected branch without having push access" [ce-5081]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5081 "Allow creating protected branches that can't be pushed to" [ce-21393]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21393 -[ee-restrict]: https://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users [perm]: ../permissions.md [ee]: https://about.gitlab.com/pricing/ + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index 26bec323f02..5cc3b8a7fc3 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Protected Tags > [Introduced][ce-10356] in GitLab 9.1. @@ -14,21 +18,21 @@ Protected tags will prevent anyone from updating or deleting the tag, as and wil To protect a tag, you need to have at least Maintainer permission level. -1. Navigate to the project's Settings -> Repository page +1. Navigate to the project's **Settings > Repository**: -  +  -1. From the **Tag** dropdown menu, select the tag you want to protect or type and click `Create wildcard`. In the screenshot below, we chose to protect all tags matching `v*`. +1. From the **Tag** dropdown menu, select the tag you want to protect or type and click **Create wildcard**. In the screenshot below, we chose to protect all tags matching `v*`: -  +  -1. From the `Allowed to create` dropdown, select who will have permission to create matching tags and then click `Protect`. +1. From the **Allowed to create** dropdown, select who will have permission to create matching tags and then click **Protect**: -  +  -1. Once done, the protected tag will appear in the "Protected tags" list. +1. Once done, the protected tag will appear in the **Protected tags** list: -  +  ## Wildcard protected tags @@ -45,13 +49,23 @@ matching the wildcard. For example: Two different wildcards can potentially match the same tag. For example, `*-stable` and `production-*` would both match a `production-stable` tag. In that case, if _any_ of these protected tags have a setting like -"Allowed to create", then `production-stable` will also inherit this setting. +**Allowed to create**, then `production-stable` will also inherit this setting. If you click on a protected tag's name, you will be presented with a list of all matching tags:  ---- +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> [ce-10356]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10356 "Protected Tags" diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 8b8aa51b6dd..647250bd02a 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -1,13 +1,18 @@ -# GitLab quick actions +--- +type: reference +--- + +# GitLab Quick Actions Quick actions are textual shortcuts for common actions on issues, epics, merge requests, and commits that are usually done by clicking buttons or dropdowns in GitLab's UI. You can enter these commands while creating a new issue or merge request, or in comments of issues, epics, merge requests, and commits. Each command should be on a separate line in order to be properly detected and executed. Once executed, -the commands are removed from the text body and not visible to anyone else. -## Quick actions for issues and merge requests +> From [GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26672), an alert is displayed when a quick action is successfully applied. + +## Quick Actions for issues and merge requests The following quick actions are applicable to both issues and merge requests threads, discussions, and descriptions: @@ -34,19 +39,21 @@ discussions, and descriptions: | `/remove_milestone` | Remove milestone | ✓ | ✓ | | `/label ~label1 ~label2` | Add label(s). Label names can also start without ~ but mixed syntax is not supported. | ✓ | ✓ | | `/unlabel ~label1 ~label2` | Remove all or specific label(s)| ✓ | ✓ | -| `/relabel ~label1 ~label2` | Replace label | ✓ | ✓ | -| <code>/copy_metadata <#issue | !merge_request></code> | Copy labels and milestone from other issue or merge request in the project | ✓ | ✓ | -| <code>/estimate <1w 3d 2h 14m></code> | Set time estimate | ✓ | ✓ | +| `/relabel ~label1 ~label2` | Replace existing label(s) with those specified | ✓ | ✓ | +| `/copy_metadata <#issue>` | Copy labels and milestone from another issue in the project | ✓ | ✓ | +| `/copy_metadata <!merge_request>` | Copy labels and milestone from another merge request in the project | ✓ | ✓ | +| `/estimate <1w 3d 2h 14m>` | Set time estimate | ✓ | ✓ | | `/remove_estimate` | Remove time estimate | ✓ | ✓ | -| <code>/spend <time(1h 30m | -1h 5m)> <date(YYYY-MM-DD)></code> | Add or subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ | +| `/spend <time(1h 30m)> <date(YYYY-MM-DD)>` | Add spent time; optionally, specify the date that time was spent on | ✓ | ✓ | +| `/spend <time(-1h 5m)> <date(YYYY-MM-DD)>` | Subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ | | `/remove_time_spent` | Remove time spent | ✓ | ✓ | -| `/lock` | Lock the discussion | ✓ | ✓ | -| `/unlock` | Unlock the discussion | ✓ | ✓ | -| <code>/due <in 2 days | this Friday | December 31st></code>| Set due date | ✓ | | +| `/lock` | Lock the thread | ✓ | ✓ | +| `/unlock` | Unlock the thread | ✓ | ✓ | +| `/due <date>` | Set due date. Examples of valid `<date>` include `in 2 days`, `this Friday` and `December 31st`. | ✓ | | | `/remove_due_date` | Remove due date | ✓ | | -| <code>/weight <0 | 1 | 2 | ...></code> | Set weight **(STARTER)** | ✓ | | +| `/weight <value>` | Set weight. Valid options for `<value>` include `0`, `1`, `2`, etc. **(STARTER)** | ✓ | | | `/clear_weight` | Clears weight **(STARTER)** | ✓ | | -| <code>/epic <&epic | group&epic | Epic URL></code> | Add to epic **(ULTIMATE)** | ✓ | | +| `/epic <epic>` | Add to epic `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. **(ULTIMATE)** | ✓ | | | `/remove_epic` | Removes from epic **(ULTIMATE)** | ✓ | | | `/promote` | Promote issue to epic **(ULTIMATE)** | ✓ | | | `/confidential` | Make confidential | ✓ | | @@ -59,6 +66,25 @@ discussions, and descriptions: | `/create_merge_request <branch name>` | Create a new merge request starting from the current issue | ✓ | | | `/relate #issue1 #issue2` | Mark issues as related **(STARTER)** | ✓ | | +## Autocomplete characters + +Many quick actions require a parameter, for example: username, milestone, and +label. [Autocomplete characters](autocomplete_characters.md) can make it easier +to enter a parameter, compared to selecting items from a list. + +## Quick actions parameters + +The easiest way to set parameters for quick actions is to use autocomplete. If +you manually enter a parameter, it must be enclosed in double quotation marks +(`"`), unless it contains only: + +1. ASCII letters. +1. Numerals. +1. Underscore, hyphen, question mark, dot, and ampersand. + +Parameters are also case-sensitive. Autocomplete handles this, and the insertion +of quotation marks, automatically. + ## Quick actions for commit messages The following quick actions are applicable for commit messages: @@ -85,8 +111,20 @@ The following quick actions are applicable for epics threads and description: | `/award :emoji:` | Toggle emoji award | | `/label ~label1 ~label2` | Add label(s) | | `/unlabel ~label1 ~label2` | Remove all or specific label(s) | -| `/relabel ~label1 ~label2` | Replace label | -| <code>/child_epic <&epic | group&epic | Epic URL></code> | Adds child epic to epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) | -| <code>/remove_child_epic <&epic | group&epic | Epic URL></code> | Removes child epic from epic ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) | -| <code>/parent_epic <&epic | group&epic | Epic URL></code> | Sets parent epic to epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) | -| <code>/remove_parent_epic | Removes parent epic from epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) | +| `/relabel ~label1 ~label2` | Replace existing label(s) with those specified | +| `/child_epic <epic>` | Adds child epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([Introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) **(ULTIMATE)**| +| `/remove_child_epic <epic>` | Removes child epic from `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([Introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ee/issues/7330)) **(ULTIMATE)** | +| `/parent_epic <epic>` | Sets parent epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic` or `epic-URL`. ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) **(ULTIMATE)** | +| `/remove_parent_epic` | Removes parent epic from epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-ee/issues/10556)) | + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/releases/img/releases.png b/doc/user/project/releases/img/releases.png Binary files differindex f8b1b7305ad..da5bcd9d913 100644 --- a/doc/user/project/releases/img/releases.png +++ b/doc/user/project/releases/img/releases.png diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 00a4f6c6a6b..aae253467f0 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -1,3 +1,7 @@ +--- +type: reference, howto +--- + # Releases > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/41766) in GitLab 11.7. @@ -60,3 +64,15 @@ Navigate to **Project > Releases** in order to see the list of releases for a gi project.  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/img/compare_branches.png b/doc/user/project/repository/branches/img/compare_branches.png Binary files differindex 52d5c518c45..52d5c518c45 100644 --- a/doc/user/project/repository/img/compare_branches.png +++ b/doc/user/project/repository/branches/img/compare_branches.png diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index a81c9197ec1..a864665602e 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -1,17 +1,41 @@ +--- +type: concepts, howto +--- + # Branches -Read through GiLab's branching documentation: +A branch is a version of a project's working tree. You create a branch for each +set of related changes you make. This keeps each set of changes separate from +each other, allowing changes to be made in parallel, without affecting each +other. + +After pushing your changes to a new branch, you can: + +- Create a [merge request](../../merge_requests/index.md) +- Perform inline code review +- [Discuss](../../../discussions/index.md) your implementation with your team +- Preview changes submitted to a new branch with [Review Apps](../../../../ci/review_apps/index.md). + +With [GitLab Starter](https://about.gitlab.com/pricing/), you can also request +[approval](../../merge_requests/merge_request_approvals.md) from your managers. + +For more information on managing branches using the GitLab UI, see: + +- [Default branches](#default-branch) +- [Create a branch](../web_editor.md#create-a-new-branch) +- [Protected branches](../../protected_branches.md#protected-branches) +- [Delete merged branches](#delete-merged-branches) +- [Branch filter search box](#branch-filter-search-box) + +You can also manage branches using the +[command line](../../../../gitlab-basics/start-using-git.md#create-a-branch). -- [Create a branch](../web_editor.md#create-a-new-branch). -- [Default branch](#default-branch). -- [Protected branches](../../protected_branches.md#protected-branches). -- [Delete merged branches](#delete-merged-branches). -- [Branch filter search box](#branch-filter-search-box). +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i>Watch the video [GitLab Flow](https://www.youtube.com/watch?v=InKNIvky2KE). See also: - [Branches API](../../../../api/branches.md), for information on operating on repository branches using the GitLab API. -- [GitLab Flow](../../../../university/training/gitlab_flow.md). Use the best of GitLab for your branching strategies. +- [GitLab Flow](../../../../university/training/gitlab_flow.md) documentation. - [Getting started with Git](../../../../topics/git/index.md) and GitLab. ## Default branch @@ -29,6 +53,17 @@ The default branch is also protected against accidental deletion. Read through the documentation on [protected branches](../../protected_branches.md#protected-branches) to learn more. +## Compare + +To compare branches in a repository: + +1. Navigate to your project's repository. +1. Select **Repository > Compare** in the sidebar. +1. Select branches to compare using the [branch filter search box](#branch-filter-search-box) +1. Click **Compare** to view the changes inline: + + + ## Delete merged branches > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6449) in GitLab 8.14. @@ -57,3 +92,15 @@ Sometimes when you have hundreds of branches you may want a more flexible matchi - `^feature` will only match branch names that begin with 'feature'. - `feature$` will only match branch names that end with 'feature'. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index cf0a986887e..b929c6a681a 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -1,38 +1,39 @@ -# Signing commits with GPG +--- +type: concepts, howto +--- -NOTE: **Note:** -The term GPG is used for all OpenPGP/PGP/GPG related material and -implementations. +# Signing commits with GPG > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9546) in GitLab 9.5. > - Subkeys support was added in GitLab 10.1. -GitLab can show whether a commit is verified or not when signed with a GPG key. -All you need to do is upload the public GPG key in your profile settings. - -GPG verified tags are not supported yet. +You can use a GPG key to sign Git commits made in a GitLab repository. Signed +commits are labeled **Verified** if the identity of the committer can be +verified. To verify the identity of a committer, GitLab requires their public +GPG key. -## Getting started with GPG +NOTE: **Note:** +The term GPG is used for all OpenPGP/PGP/GPG related material and +implementations. -Here are a few guides to get you started with GPG: +GPG verified tags are not supported yet. -- [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) -- [Managing OpenPGP Keys](https://riseup.net/en/security/message-security/openpgp/gpg-keys) -- [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) -- [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) +See the [further reading](#further-reading) section for more details on GPG. ## How GitLab handles GPG GitLab uses its own keyring to verify the GPG signature. It does not access any public key server. -In order to have a commit verified on GitLab the corresponding public key needs -to be uploaded to GitLab. For a signature to be verified three conditions need -to be met: +For a commit to be verified by GitLab: -1. The public key needs to be added your GitLab account -1. One of the emails in the GPG key matches a **verified** email address you use in GitLab -1. The committer's email matches the verified email from the gpg key +- The committer must have a GPG public/private key pair. +- The committer's public key must have been uploaded to their GitLab + account. +- One of the emails in the GPG key must match a **verified** email address + used by the committer in GitLab. +- The committer's email address must match the verified email address from the + GPG key. ## Generating a GPG key @@ -45,94 +46,94 @@ started: 1. Generate the private/public key pair with the following command, which will spawn a series of questions: - ```sh - gpg --full-gen-key - ``` + ```sh + gpg --full-gen-key + ``` - NOTE: **Note:** - In some cases like Gpg4win on Windows and other macOS versions, the command - here may be `gpg --gen-key`. + NOTE: **Note:** + In some cases like Gpg4win on Windows and other macOS versions, the command + here may be `gpg --gen-key`. 1. The first question is which algorithm can be used. Select the kind you want or press <kbd>Enter</kbd> to choose the default (RSA and RSA): - ``` - Please select what kind of key you want: - (1) RSA and RSA (default) - (2) DSA and Elgamal - (3) DSA (sign only) - (4) RSA (sign only) - Your selection? 1 - ``` - -1. The next question is key length. We recommend to choose the highest value - which is `4096`: - - ``` - RSA keys may be between 1024 and 4096 bits long. - What keysize do you want? (2048) 4096 - Requested keysize is 4096 bits - ``` -1. Next, you need to specify the validity period of your key. This is something - subjective, and you can use the default value which is to never expire: - - ``` - Please specify how long the key should be valid. - 0 = key does not expire - <n> = key expires in n days - <n>w = key expires in n weeks - <n>m = key expires in n months - <n>y = key expires in n years - Key is valid for? (0) 0 - Key does not expire at all - ``` + ``` + Please select what kind of key you want: + (1) RSA and RSA (default) + (2) DSA and Elgamal + (3) DSA (sign only) + (4) RSA (sign only) + Your selection? 1 + ``` + +1. The next question is key length. We recommend you choose `4096`: + + ``` + RSA keys may be between 1024 and 4096 bits long. + What keysize do you want? (2048) 4096 + Requested keysize is 4096 bits + ``` + +1. Specify the validity period of your key. This is something + subjective, and you can use the default value, which is to never expire: + + ``` + Please specify how long the key should be valid. + 0 = key does not expire + <n> = key expires in n days + <n>w = key expires in n weeks + <n>m = key expires in n months + <n>y = key expires in n years + Key is valid for? (0) 0 + Key does not expire at all + ``` 1. Confirm that the answers you gave were correct by typing `y`: - ``` - Is this correct? (y/N) y - ``` + ``` + Is this correct? (y/N) y + ``` -1. Enter you real name, the email address to be associated with this key (should - match a verified email address you use in GitLab) and an optional comment - (press <kbd>Enter</kbd> to skip): +1. Enter your real name, the email address to be associated with this key + (should match a verified email address you use in GitLab) and an optional + comment (press <kbd>Enter</kbd> to skip): - ``` - GnuPG needs to construct a user ID to identify your key. + ``` + GnuPG needs to construct a user ID to identify your key. - Real name: Mr. Robot - Email address: <your_email> - Comment: - You selected this USER-ID: - "Mr. Robot <your_email>" + Real name: Mr. Robot + Email address: <your_email> + Comment: + You selected this USER-ID: + "Mr. Robot <your_email>" - Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O - ``` + Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O + ``` 1. Pick a strong password when asked and type it twice to confirm. 1. Use the following command to list the private GPG key you just created: - ``` - gpg --list-secret-keys --keyid-format LONG <your_email> - ``` + ``` + gpg --list-secret-keys --keyid-format LONG <your_email> + ``` - Replace `<your_email>` with the email address you entered above. + Replace `<your_email>` with the email address you entered above. 1. Copy the GPG key ID that starts with `sec`. In the following example, that's `30F2B65B9246B6CA`: - ``` - sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] - D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA - uid [ultimate] Mr. Robot <your_email> - ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] - ``` + ``` + sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] + D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA + uid [ultimate] Mr. Robot <your_email> + ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] + ``` 1. Export the public key of that ID (replace your key ID from the previous step): - ``` - gpg --armor --export 30F2B65B9246B6CA - ``` + ``` + gpg --armor --export 30F2B65B9246B6CA + ``` 1. Finally, copy the public key and [add it in your profile settings](#adding-a-gpg-key-to-your-account) @@ -146,17 +147,17 @@ You can add a GPG key in your profile's settings: 1. On the upper right corner, click on your avatar and go to your **Settings**. -  +  1. Navigate to the **GPG keys** tab and paste your _public_ key in the 'Key' box. -  +  1. Finally, click on **Add key** to add it to GitLab. You will be able to see its fingerprint, the corresponding email address and creation date. -  +  ## Associating your GPG key with Git @@ -166,29 +167,29 @@ key to use. 1. Use the following command to list the private GPG key you just created: - ```sh - gpg --list-secret-keys --keyid-format LONG <your_email> - ``` + ```sh + gpg --list-secret-keys --keyid-format LONG <your_email> + ``` - Replace `<your_email>` with the email address you entered above. + Replace `<your_email>` with the email address you entered above. 1. Copy the GPG key ID that starts with `sec`. In the following example, that's `30F2B65B9246B6CA`: - ``` - sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] - D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA - uid [ultimate] Mr. Robot <your_email> - ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] - ``` + ``` + sec rsa4096/30F2B65B9246B6CA 2017-08-18 [SC] + D5E4F29F3275DC0CDA8FFC8730F2B65B9246B6CA + uid [ultimate] Mr. Robot <your_email> + ssb rsa4096/B7ABC0813E4028C0 2017-08-18 [E] + ``` 1. Tell Git to use that key to sign the commits: - ```sh - git config --global user.signingkey 30F2B65B9246B6CA - ``` + ```sh + git config --global user.signingkey 30F2B65B9246B6CA + ``` - Replace `30F2B65B9246B6CA` with your GPG key ID. + Replace `30F2B65B9246B6CA` with your GPG key ID. 1. (Optional) If Git is using `gpg` and you get errors like `secret key not available` or `gpg: signing failed: secret key not available`, run the following command to @@ -206,9 +207,9 @@ commits: 1. Commit like you used to, the only difference is the addition of the `-S` flag: - ``` - git commit -S -m "My commit msg" - ``` + ``` + git commit -S -m "My commit msg" + ``` 1. Enter the passphrase of your GPG key when asked. 1. Push to GitLab and check that your commits [are verified](#verifying-commits). @@ -227,13 +228,13 @@ git config --global commit.gpgsign true "Verified" or "Unverified", depending on the verification status of the GPG signature. -  +  1. By clicking on the GPG badge, details of the signature are displayed. -  +  -  +  ## Revoking a GPG key @@ -269,3 +270,24 @@ via [push rules](../../../../push_rules/push_rules.md). ## GPG signing API Learn how to [get the GPG signature from a commit via API](../../../../api/commits.md#get-gpg-signature-of-a-commit). + +## Further reading + +For more details about GPG, see: + +- [Git Tools - Signing Your Work](https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work) +- [Managing OpenPGP Keys](https://riseup.net/en/security/message-security/openpgp/gpg-keys) +- [OpenPGP Best Practices](https://riseup.net/en/security/message-security/openpgp/best-practices) +- [Creating a new GPG key with subkeys](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) (advanced) + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/img/download_source_code.png b/doc/user/project/repository/img/download_source_code.png Binary files differindex 17f2cb4b3e8..56808061980 100644 --- a/doc/user/project/repository/img/download_source_code.png +++ b/doc/user/project/repository/img/download_source_code.png diff --git a/doc/user/project/repository/img/repository_languages.png b/doc/user/project/repository/img/repository_languages.png Binary files differdeleted file mode 100644 index 5977ad7faae..00000000000 --- a/doc/user/project/repository/img/repository_languages.png +++ /dev/null diff --git a/doc/user/project/repository/img/repository_languages_v12_2.gif b/doc/user/project/repository/img/repository_languages_v12_2.gif Binary files differnew file mode 100644 index 00000000000..d0a0e57c919 --- /dev/null +++ b/doc/user/project/repository/img/repository_languages_v12_2.gif diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 5b5685b3418..bd966185c94 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -1,3 +1,7 @@ +--- +type: concepts, howto +--- + # Repository A [repository](https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository) @@ -111,33 +115,7 @@ GitLab. ## Branches -When you submit changes in a new [branch](branches/index.md), you create a new version -of that project's file tree. Your branch contains all the changes -you are presenting, which are detected by Git line by line. - -To continue your workflow, once you pushed your changes to a new branch, -you can create a [merge request](../merge_requests/index.md), perform -inline code review, and [discuss](../../discussions/index.md) -your implementation with your team. -You can live preview changes submitted to a new branch with -[Review Apps](../../../ci/review_apps/index.md). - -With [GitLab Starter](https://about.gitlab.com/pricing/), you can also request -[approval](../merge_requests/merge_request_approvals.md) from your managers. - -To create, delete, and view [branches](branches/index.md) via GitLab's UI: - -- [Default branches](branches/index.md#default-branch) -- [Create a branch](web_editor.md#create-a-new-branch) -- [Protected branches](../protected_branches.md#protected-branches) -- [Delete merged branches](branches/index.md#delete-merged-branches) -- [Branch filter search box](branches/index.md#branch-filter-search-box) - -Alternatively, you can use the -[command line](../../../gitlab-basics/start-using-git.md#create-a-branch). - -To learn more about branching strategies read through the -[GitLab Flow](../../../university/training/gitlab_flow.md) documentation. +For details, see [Branches](branches/index.md). ## Commits @@ -204,7 +182,7 @@ were used and display this on the projects pages. If this information is missing be added after updating the default branch on the project. This process can take up to 5 minutes. - + Not all files are detected, among others; documentation, vendored code, and most markup languages are excluded. This behaviour can be @@ -213,14 +191,6 @@ detected, add the following to `.gitattributes` in the root of your repository. > *.proto linguist-detectable=true -## Compare - -Select branches to compare using the [branch filter search box](branches/index.md#branch-filter-search-box), then click the **Compare** button to view the changes inline: - - - -Find it under your project's **Repository > Compare**. - ## Locked files **(PREMIUM)** Use [File Locking](../file_lock.md) to @@ -256,3 +226,15 @@ By clicking the download icon, a dropdown will open with links to download the f `tar`, `tar.gz`, and `tar.bz2`. - **Artifacts:** allows users to download the artifacts of the latest CI build. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 7765a3d7438..3adf66e4b6f 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -1,3 +1,7 @@ +--- +type: howto +--- + # Reducing the repository size using Git A GitLab Enterprise Edition administrator can set a [repository size limit](../../admin_area/settings/account_and_limit_settings.md) @@ -54,50 +58,50 @@ removed from the repository. 1. Navigate to your repository: - ``` - cd my_repository/ - ``` + ``` + cd my_repository/ + ``` 1. Change to the branch you want to remove the big file from: - ``` - git checkout master - ``` + ``` + git checkout master + ``` 1. Create a commit removing the large file from the branch, if it still exists: - ``` - git rm path/to/big_file.mpg - git commit -m 'Remove unneeded large file' - ``` + ``` + git rm path/to/big_file.mpg + git commit -m 'Remove unneeded large file' + ``` 1. Rewrite history: - ``` - bfg --delete-files path/to/big_file.mpg - ``` + ``` + bfg --delete-files path/to/big_file.mpg + ``` - An object map file will be written to `object-id-map.old-new.txt`. Keep it - around - you'll need it for the final step! + An object map file will be written to `object-id-map.old-new.txt`. Keep it + around - you'll need it for the final step! 1. Force-push the changes to GitLab: - ``` - git push --force-with-lease origin master - ``` + ``` + git push --force-with-lease origin master + ``` - If this step fails, someone has changed the `master` branch while you were - rewriting history. You could restore the branch and re-run BFG to preserve - their changes, or use `git push --force` to overwrite their changes. + If this step fails, someone has changed the `master` branch while you were + rewriting history. You could restore the branch and re-run BFG to preserve + their changes, or use `git push --force` to overwrite their changes. 1. Navigate to **Project > Settings > Repository > Repository Cleanup**: -  +  - Upload the `object-id-map.old-new.txt` file and press **Start cleanup**. - This will remove any internal git references to the old commits, and run - `git gc` against the repository. You will receive an email once it has - completed. + Upload the `object-id-map.old-new.txt` file and press **Start cleanup**. + This will remove any internal git references to the old commits, and run + `git gc` against the repository. You will receive an email once it has + completed. NOTE: **Note:** This process will remove some copies of the rewritten commits from GitLab's @@ -110,32 +114,44 @@ purposes! 1. Navigate to your repository: - ``` - cd my_repository/ - ``` + ``` + cd my_repository/ + ``` 1. Change to the branch you want to remove the big file from: - ``` - git checkout master - ``` + ``` + git checkout master + ``` 1. Use `filter-branch` to remove the big file: - ``` - git filter-branch --force --tree-filter 'rm -f path/to/big_file.mpg' HEAD - ``` + ``` + git filter-branch --force --tree-filter 'rm -f path/to/big_file.mpg' HEAD + ``` 1. Instruct Git to purge the unwanted data: - ``` - git reflog expire --expire=now --all && git gc --prune=now --aggressive - ``` + ``` + git reflog expire --expire=now --all && git gc --prune=now --aggressive + ``` 1. Lastly, force push to the repository: - ``` - git push --force origin master - ``` + ``` + git push --force origin master + ``` Your repository should now be below the size limit. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 253e5374f52..b5299eaca27 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -1,3 +1,7 @@ +--- +type: howto +--- + # GitLab Web Editor Sometimes it's easier to make quick changes directly from the GitLab interface @@ -13,8 +17,6 @@ Choose **New file** from the dropdown.  ---- - Enter a file name in the **File name** box. Then, add file content in the editor area. Add a descriptive commit message and choose a branch. The branch field will default to the branch you were viewing in the file browser. If you enter @@ -59,8 +61,6 @@ selector. Choose **Upload file** from the dropdown.  ---- - Once the upload dialog pops up there are two ways to upload your file. Either drag and drop a file on the pop up or use the **click to upload** link. A file preview will appear once you have selected a file to upload. @@ -80,8 +80,6 @@ Choose **New directory** from the dropdown.  ---- - In the new directory dialog enter a directory name, a commit message and choose the target branch. Click **Create directory** to finish. @@ -107,7 +105,7 @@ Once you click it, a new branch will be created that diverges from the default branch of your project, by default `master`. The branch name will be based on the title of the issue and as a prefix, it will have its internal ID. Thus, the example screenshot above will yield a branch named -`2-et-cum-et-sed-expedita-repellat-consequatur-ut-assumenda-numquam-rerum`. +`23177-add-support-for-rich-references-to-referables`. Since GitLab 9.0, when you click the `New branch` in an empty repository project, GitLab automatically creates the master branch, commits a blank `README.md` file to it and creates and redirects you to a new branch based on the issue title. If your [project is already configured with a deployment service][project-services-doc] (e.g. Kubernetes), GitLab takes one step further and prompts you to set up [auto deploy][auto-deploy-doc] by helping you create a `.gitlab-ci.yml` file. @@ -129,8 +127,6 @@ choose **New branch** from the dropdown.  ---- - Enter a new **Branch name**. Optionally, change the **Create from** field to choose which branch, tag or commit SHA this new branch will originate from. This field will autocomplete if you start typing an existing branch or tag. @@ -139,8 +135,6 @@ branch.  ---- - You can now make changes to any files, as needed. When you're ready to merge the changes back to master you can use the widget at the top of the screen. This widget only appears for a period of time after you create the branch or @@ -156,8 +150,6 @@ SHA. From a project's files page, choose **New tag** from the dropdown.  ---- - Give the tag a name such as `v1.0.0`. Choose the branch or SHA from which you would like to create this new tag. You can optionally add a message and release notes. The release notes section supports markdown format and you can @@ -181,3 +173,15 @@ through the web editor, you can choose to use another of your linked email addresses from the **User Settings > Edit Profile** page. [ce-2808]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2808 + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index f899120f0c2..645b9bffcc1 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -53,36 +53,35 @@ Service Desk is enabled on GitLab.com. If you're a [Silver subscriber](https://about.gitlab.com/gitlab-com/), you can skip the step 1 below; you only need to enable it per project. -1. [Set up incoming email](../../administration/incoming_email.md#set-it-up) for the GitLab instance. This must - support [email sub-addressing](../../administration/incoming_email.md#email-sub-addressing). -2. Navigate to your project's **Settings** and scroll down to the **Service Desk** - section. -3. If you have the correct access and an Premium license, - you will see an option to set up Service Desk: +1. [Set up incoming email](../../administration/incoming_email.md#set-it-up) for the GitLab instance. This must + support [email sub-addressing](../../administration/incoming_email.md#email-sub-addressing). +1. Navigate to your project's **Settings** and scroll down to the **Service Desk** + section. +1. If you have the correct access and an Premium license, + you will see an option to set up Service Desk: -  +  -4. Checking that box will enable Service Desk for the project, and show a - unique email address to email issues to the project. These issues will be - [confidential](issues/confidential_issues.md), so they will only be visible to project members. +1. Checking that box will enable Service Desk for the project, and show a + unique email address to email issues to the project. These issues will be + [confidential](issues/confidential_issues.md), so they will only be visible to project members. - **Warning**: this email address can be used by anyone to create an issue on - this project, whether or not they have access to your GitLab instance. - We recommend **putting this behind an alias** so that it can be changed if - needed, and **[enabling Akismet](../../integration/akismet.md)** on your GitLab instance to add spam - checking to this service. Unblocked email spam would result in many spam - issues being created, and may disrupt your GitLab service. + **Warning**: this email address can be used by anyone to create an issue on + this project, whether or not they have access to your GitLab instance. + We recommend **putting this behind an alias** so that it can be changed if + needed, and **[enabling Akismet](../../integration/akismet.md)** on your GitLab instance to add spam + checking to this service. Unblocked email spam would result in many spam + issues being created, and may disrupt your GitLab service. -  +  - _In GitLab 11.7, we updated the format of the generated email address. - However the older format is still supported, allowing existing aliases - or contacts to continue working._ + _In GitLab 11.7, we updated the format of the generated email address. + However the older format is still supported, allowing existing aliases + or contacts to continue working._ +1. Service Desk is now enabled for this project! You should be able to access it from your project's navigation **Issue submenu**: -5. Service Desk is now enabled for this project! You should be able to access it from your project's navigation **Issue submenu**: - -  +  ## Using Service Desk diff --git a/doc/user/project/settings/img/general_settings.png b/doc/user/project/settings/img/general_settings.png Binary files differindex 4ff6fff5ca3..f88a158d2be 100644 --- a/doc/user/project/settings/img/general_settings.png +++ b/doc/user/project/settings/img/general_settings.png diff --git a/doc/user/project/settings/img/import_export_download_export.png b/doc/user/project/settings/img/import_export_download_export.png Binary files differindex ab81c87bf5f..c7ab7565fc7 100644 --- a/doc/user/project/settings/img/import_export_download_export.png +++ b/doc/user/project/settings/img/import_export_download_export.png diff --git a/doc/user/project/settings/img/import_export_export_button.png b/doc/user/project/settings/img/import_export_export_button.png Binary files differindex 9e368739695..6933e3edfcc 100644 --- a/doc/user/project/settings/img/import_export_export_button.png +++ b/doc/user/project/settings/img/import_export_export_button.png diff --git a/doc/user/project/settings/img/import_export_mail_link.png b/doc/user/project/settings/img/import_export_mail_link.png Binary files differindex 985c37650d3..1bd9a071178 100644 --- a/doc/user/project/settings/img/import_export_mail_link.png +++ b/doc/user/project/settings/img/import_export_mail_link.png diff --git a/doc/user/project/settings/img/import_export_new_project.png b/doc/user/project/settings/img/import_export_new_project.png Binary files differindex fc1f73c5d6e..0e2365ecb68 100644 --- a/doc/user/project/settings/img/import_export_new_project.png +++ b/doc/user/project/settings/img/import_export_new_project.png diff --git a/doc/user/project/settings/img/import_export_select_file.png b/doc/user/project/settings/img/import_export_select_file.png Binary files differindex e3e1a5ef980..90a3e8d5c4e 100644 --- a/doc/user/project/settings/img/import_export_select_file.png +++ b/doc/user/project/settings/img/import_export_select_file.png diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 98bcc7cc09f..9c1a31fb7c3 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -2,33 +2,33 @@ >**Notes:** > -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/3050) in GitLab 8.9. -> - Importing will not be possible if the import instance version differs from -> that of the exporter. -> - For GitLab admins, please read through -> [Project import/export administration](../../../administration/raketasks/project_import_export.md). -> - For existing installations, the project import option has to be enabled in -> application settings (`/admin/application_settings`) under 'Import sources'. -> Ask your administrator if you don't see the **GitLab export** button when -> creating a new project. -> - Starting with GitLab 10.0, administrators can disable the project export option -> on the GitLab instance in application settings (`/admin/application_settings`) -> under 'Visibility and Access Controls'. -> - You can find some useful raketasks if you are an administrator in the -> [import_export](../../../administration/raketasks/project_import_export.md) raketask. -> - The exports are stored in a temporary [shared directory](../../../development/shared_files.md) -> and are deleted every 24 hours by a specific worker. -> - Group members will get exported as project members, as long as the user has -> maintainer or admin access to the group where the exported project lives. An admin -> in the import side is required to map the users, based on email or username. -> Otherwise, a supplementary comment is left to mention the original author and -> the MRs, notes or issues will be owned by the importer. -> - Project members with owner access will get imported as maintainers. -> - Control project Import/Export with the [API](../../../api/project_import_export.md). -> - If an imported project contains merge requests originated from forks, -> then new branches associated with such merge requests will be created -> within a project during the import/export. Thus, the number of branches -> in the exported project could be bigger than in the original project. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/3050) in GitLab 8.9. +> - Importing will not be possible if the import instance version differs from +> that of the exporter. +> - For GitLab admins, please read through +> [Project import/export administration](../../../administration/raketasks/project_import_export.md). +> - For existing installations, the project import option has to be enabled in +> application settings (`/admin/application_settings`) under 'Import sources'. +> Ask your administrator if you don't see the **GitLab export** button when +> creating a new project. +> - Starting with GitLab 10.0, administrators can disable the project export option +> on the GitLab instance in application settings (`/admin/application_settings`) +> under 'Visibility and Access Controls'. +> - You can find some useful raketasks if you are an administrator in the +> [import_export](../../../administration/raketasks/project_import_export.md) raketask. +> - The exports are stored in a temporary [shared directory](../../../development/shared_files.md) +> and are deleted every 24 hours by a specific worker. +> - Group members will get exported as project members, as long as the user has +> maintainer or admin access to the group where the exported project lives. An admin +> in the import side is required to map the users, based on email or username. +> Otherwise, a supplementary comment is left to mention the original author and +> the MRs, notes or issues will be owned by the importer. +> - Project members with owner access will get imported as maintainers. +> - Control project Import/Export with the [API](../../../api/project_import_export.md). +> - If an imported project contains merge requests originated from forks, +> then new branches associated with such merge requests will be created +> within a project during the import/export. Thus, the number of branches +> in the exported project could be bigger than in the original project. Existing projects running on any GitLab instance or GitLab.com can be exported with all their related data and be moved into a new GitLab instance. @@ -66,6 +66,7 @@ The following items will be exported: - Issues with comments, merge requests with diffs and comments, labels, milestones, snippets, and other project entities - LFS objects +- Issue boards The following items will NOT be exported: @@ -79,7 +80,7 @@ The following items will NOT be exported: - Awards NOTE: **Note:** -For more details on the specific data persisted in a project export, see the +For more details on the specific data persisted in a project export, see the [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/import_export/import_export.yml) file. ## Exporting a project and its data @@ -90,29 +91,34 @@ For more details on the specific data persisted in a project export, see the 1. Scroll down to find the **Export project** button: -  +  1. Once the export is generated, you should receive an e-mail with a link to download the file: -  +  1. Alternatively, you can come back to the project settings and download the file from there, or generate a new export. Once the file available, the page should show the **Download export** button: -  +  ## Importing the project -1. The GitLab project import feature is the first import option when creating a +1. The GitLab project import feature is the first import option when creating a new project. Click on **GitLab export**: -  +  1. Enter your project name and URL. Then select the file you exported previously: -  +  1. Click on **Import project** to begin importing. Your newly imported project page will appear soon. + +NOTE: **Note:** +If use of the `Internal` visibility level +[is restricted](../../../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects), +all imported projects are given the visibility of `Private`. diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 06b431decad..4e3db95b6d6 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -32,6 +32,12 @@ links will be missing from the sidebar UI. You can still access them with direct links if you can access Merge Requests. This is deliberate, if you can see Issues or Merge Requests, both of which use Labels and Milestones, then you shouldn't be denied access to Labels and Milestones pages. +#### Disabling email notifications + +You can disable all email notifications related to the project by selecting the +**Disable email notifications** checkbox. Only the project owner is allowed to change +this setting. + ### Issue settings Add an [issue description template](../description_templates.md#description-templates) to your project, so that every new issue will start with a custom template. @@ -44,7 +50,7 @@ Set up your project's merge request settings: - Merge request [description templates](../description_templates.md#description-templates). - Enable [merge request approvals](../merge_requests/merge_request_approvals.md). **(STARTER)** - Enable [merge only of pipeline succeeds](../merge_requests/merge_when_pipeline_succeeds.md). -- Enable [merge only when all discussions are resolved](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved). +- Enable [merge only when all discussions are resolved](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-threads-are-resolved).  @@ -96,7 +102,7 @@ To rename a repository: 1. Hit **Rename project**. Remember that this can have unintended side effects since everyone with the -old URL will not be able to push or pull. Read more about what happens with the +old URL will not be able to push or pull. Read more about what happens with the [redirects when renaming repositories](../index.md#redirects-when-changing-repository-paths). #### Transferring an existing project into another namespace diff --git a/doc/user/project/web_ide/img/terminal_status.png b/doc/user/project/web_ide/img/terminal_status.png Binary files differindex c37aa02b07a..91c341a9854 100644 --- a/doc/user/project/web_ide/img/terminal_status.png +++ b/doc/user/project/web_ide/img/terminal_status.png diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 3d92508ad04..9bf400e7dff 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -295,6 +295,5 @@ active terminal at a time. connect to the runner. Please try to stop and restart the terminal. If the problem persists, double check your runner configuration. - [ce]: https://about.gitlab.com/pricing/ [ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index e3c6cd6d6ff..d1bab47de25 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -28,11 +28,12 @@ NOTE: **Note:** Requires Developer [permissions](../../permissions.md). Create a new page by clicking the **New page** button that can be found -in all wiki pages. You will be asked to fill in the page name from which GitLab -will create the path to the page. You can specify a full path for the new file -and any missing directories will be created automatically. +in all wiki pages. - +You will be asked to fill in a title for your new wiki page. Wiki titles +also determine the path to the wiki page. You can specify a full path +(using "`/`" for subdirectories) for the new title and any missing +directories will be created automatically. Once you enter the page name, it's time to fill in its content. GitLab wikis support Markdown, RDoc and AsciiDoc. For Markdown based pages, all the diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 9e8475d8294..68532ccee65 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -40,19 +40,18 @@ It is currently not possible to create a project with the following names: Currently the following names are reserved as top level groups: -- 503.html - \- - .well-known - 404.html - 422.html - 500.html - 502.html +- 503.html - abuse_reports - admin - api - apple-touch-icon-precomposed.png - apple-touch-icon.png -- files - assets - autocomplete - ci @@ -61,12 +60,14 @@ Currently the following names are reserved as top level groups: - explore - favicon.ico - favicon.png +- files - groups - health_check - help - import - invites - jwt +- login - notification_settings - oauth - profile @@ -78,8 +79,6 @@ Currently the following names are reserved as top level groups: - sent_notifications - slash-command-logo.png - snippets -- u -- unicorn_test - unsubscribes - uploads - users @@ -88,22 +87,5 @@ Currently the following names are reserved as top level groups: These group names are unavailable as subgroup names: - \- -- activity -- analytics -- audit_events -- avatar -- edit -- group_members -- hooks -- issues -- labels -- ldap -- ldap_group_links -- merge_requests -- milestones -- notification_setting -- pipeline_quota -- projects -- subgroups [reserved]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/path_regex.rb diff --git a/doc/user/search/advanced_search_syntax.md b/doc/user/search/advanced_search_syntax.md index ad9f065b19b..c3d22e4fd29 100644 --- a/doc/user/search/advanced_search_syntax.md +++ b/doc/user/search/advanced_search_syntax.md @@ -50,9 +50,9 @@ here's a quick guide: The Advanced Syntax Search also supports the use of filters. The available filters are: - - filename: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. - - path: Filters by path. You can use the glob (`*`) operator for fuzzy matching. - - extension: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. +- filename: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. +- path: Filters by path. You can use the glob (`*`) operator for fuzzy matching. +- extension: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. To use them, simply add them to your query in the format `<filter_name>:<value>` without any spaces between the colon (`:`) and the value. diff --git a/doc/user/search/index.md b/doc/user/search/index.md index c34b9ae3d7e..8d7b4a429aa 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -55,12 +55,12 @@ Selecting **Any** does the opposite. It returns results that have a non-empty va You can filter issues and merge requests by specific terms included in titles or descriptions. - Syntax - - Searches look for all the words in a query, in any order. E.g.: searching - issues for `display bug` will return all issues matching both those words, in any order. - - To find the exact term, use double quotes: `"display bug"` + - Searches look for all the words in a query, in any order. E.g.: searching + issues for `display bug` will return all issues matching both those words, in any order. + - To find the exact term, use double quotes: `"display bug"` - Limitation - - For performance reasons, terms shorter than 3 chars are ignored. E.g.: searching - issues for `included in titles` is same as `included titles` + - For performance reasons, terms shorter than 3 chars are ignored. E.g.: searching + issues for `included in titles` is same as `included titles`  diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 6ad61932868..c6396672e59 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -38,7 +38,7 @@ comments: false - [Authorization for merge requests](../user/project/merge_requests/authorization_for_merge_requests.md) - [Cherry-pick changes](../user/project/merge_requests/cherry_pick_changes.md) - [Merge when pipeline succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md) - - [Resolve discussion comments in merge requests reviews](../user/discussions/index.md) + - [Resolve threads in merge requests reviews](../user/discussions/index.md) - [Resolve merge conflicts in the UI](../user/project/merge_requests/resolve_conflicts.md) - [Revert changes in the UI](../user/project/merge_requests/revert_changes.md) - [Merge requests versions](../user/project/merge_requests/versions.md) diff --git a/doc/workflow/file_finder.md b/doc/workflow/file_finder.md index 8d87b030c83..52c83caeea8 100644 --- a/doc/workflow/file_finder.md +++ b/doc/workflow/file_finder.md @@ -2,8 +2,6 @@ > [Introduced][gh-9889] in GitLab 8.4. ---- - The file finder feature allows you to quickly shortcut your way when you are searching for a file in a repository using the GitLab UI. @@ -12,8 +10,6 @@ project.  ---- - For those who prefer to keep their fingers on the keyboard, there is a [shortcut button](shortcuts.md) as well, which you can invoke from _anywhere_ in a project. diff --git a/doc/workflow/forking/branch_select.png b/doc/workflow/forking/branch_select.png Binary files differindex 77236137190..0ea5410f832 100644 --- a/doc/workflow/forking/branch_select.png +++ b/doc/workflow/forking/branch_select.png diff --git a/doc/workflow/forking/merge_request.png b/doc/workflow/forking/merge_request.png Binary files differindex 407ddfb4799..43851203f3f 100644 --- a/doc/workflow/forking/merge_request.png +++ b/doc/workflow/forking/merge_request.png diff --git a/doc/workflow/forking_workflow.md b/doc/workflow/forking_workflow.md index 02be0ad191d..82c2709143d 100644 --- a/doc/workflow/forking_workflow.md +++ b/doc/workflow/forking_workflow.md @@ -10,31 +10,24 @@ document more information about using branches to work together. Forking a project is in most cases a two-step process. -1. Click on the fork button located in the middle of the page or a project's - home page right next to the stars button. +1. Click on the fork button located located in between the star and clone buttons on the project's home page. -  +  - --- +1. Once you do that, you'll be presented with a screen where you can choose + the namespace to fork to. Only namespaces (groups and your own + namespace) where you have write access to, will be shown. Click on the + namespace to create your fork there. -1. Once you do that, you'll be presented with a screen where you can choose - the namespace to fork to. Only namespaces (groups and your own - namespace) where you have write access to, will be shown. Click on the - namespace to create your fork there. +  -  + **Note:** + If the namespace you chose to fork the project to has another project with + the same path name, you will be presented with a warning that the forking + could not be completed. Try to resolve the error before repeating the forking + process. - --- - - **Note:** - If the namespace you chose to fork the project to has another project with - the same path name, you will be presented with a warning that the forking - could not be completed. Try to resolve the error and repeat the forking - process. - -  - - --- +  After the forking is done, you can start working on the newly created repository. There, you will have full [Owner](../user/permissions.md) @@ -50,7 +43,7 @@ create the [merge request](merge_requests.md).  You can then assign the merge request to someone to have them review -your changes. Upon pressing the 'Accept Merge Request' button, your +your changes. Upon pressing the 'Submit Merge Request' button, your changes will be added to the repository and branch you're merging into.  diff --git a/doc/workflow/git_annex.md b/doc/workflow/git_annex.md index 84d25951908..cdc38a5b7eb 100644 --- a/doc/workflow/git_annex.md +++ b/doc/workflow/git_annex.md @@ -54,13 +54,13 @@ sudo yum install epel-release && sudo yum install git-annex For omnibus-gitlab packages, only one configuration setting is needed. The Omnibus package will internally set the correct options in all locations. -1. In `/etc/gitlab/gitlab.rb` add the following line: +1. In `/etc/gitlab/gitlab.rb` add the following line: - ```ruby - gitlab_shell['git_annex_enabled'] = true - ``` + ```ruby + gitlab_shell['git_annex_enabled'] = true + ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab][] for the changes to take effect. ### Configuration for installations from source @@ -69,20 +69,20 @@ There are 2 settings to enable git-annex on your GitLab server. One is located in `config/gitlab.yml` of the GitLab repository and the other one is located in `config.yml` of gitlab-shell. -1. In `config/gitlab.yml` add or edit the following lines: +1. In `config/gitlab.yml` add or edit the following lines: - ```yaml - gitlab_shell: - git_annex_enabled: true - ``` + ```yaml + gitlab_shell: + git_annex_enabled: true + ``` -1. In `config.yml` of gitlab-shell add or edit the following lines: +1. In `config.yml` of gitlab-shell add or edit the following lines: - ```yaml - git_annex_enabled: true - ``` + ```yaml + git_annex_enabled: true + ``` -1. Save the files and [restart GitLab][] for the changes to take effect. +1. Save the files and [restart GitLab][] for the changes to take effect. ## Using GitLab git-annex @@ -200,7 +200,7 @@ searching for your distribution. Although there is no general guide for `git-annex` errors, there are a few tips on how to go around the warnings. -### git-annex-shell: Not a git-annex or gcrypt repository. +### git-annex-shell: Not a git-annex or gcrypt repository This warning can appear on the initial `git annex sync --content` and is caused by differences in `git-annex-shell`. You can read more about it diff --git a/doc/workflow/img/forking_workflow_choose_namespace.png b/doc/workflow/img/forking_workflow_choose_namespace.png Binary files differindex b34b12090a1..eb023ca85f2 100644 --- a/doc/workflow/img/forking_workflow_choose_namespace.png +++ b/doc/workflow/img/forking_workflow_choose_namespace.png diff --git a/doc/workflow/img/forking_workflow_fork_button.png b/doc/workflow/img/forking_workflow_fork_button.png Binary files differindex 941d5363c35..7fb07529b6d 100644 --- a/doc/workflow/img/forking_workflow_fork_button.png +++ b/doc/workflow/img/forking_workflow_fork_button.png diff --git a/doc/workflow/img/forking_workflow_path_taken_error.png b/doc/workflow/img/forking_workflow_path_taken_error.png Binary files differindex df938da5677..ef62d0ab6a9 100644 --- a/doc/workflow/img/forking_workflow_path_taken_error.png +++ b/doc/workflow/img/forking_workflow_path_taken_error.png diff --git a/doc/workflow/img/notification_global_settings.png b/doc/workflow/img/notification_global_settings.png Binary files differindex 72f7418f1f8..699f726c442 100644 --- a/doc/workflow/img/notification_global_settings.png +++ b/doc/workflow/img/notification_global_settings.png diff --git a/doc/workflow/issue_weight.md b/doc/workflow/issue_weight.md index afb623e1967..a80519f0748 100644 --- a/doc/workflow/issue_weight.md +++ b/doc/workflow/issue_weight.md @@ -9,7 +9,7 @@ value or complexity a given issue has or will cost. You can set the weight of an issue during its creation, by simply changing the value in the dropdown menu. You can set it to a non-negative integer -value from 0, 1, 2, and so on. (The database stores a 4-byte value, so the +value from 0, 1, 2, and so on. (The database stores a 4-byte value, so the upper bound is essentially limitless). You can remove weight from an issue as well. diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md index 03af8fad759..7e880b3009d 100644 --- a/doc/workflow/lfs/lfs_administration.md +++ b/doc/workflow/lfs/lfs_administration.md @@ -46,8 +46,7 @@ In `config/gitlab.yml`: ## Storing LFS objects in remote object storage -> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core -in 10.7. +> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7. It is possible to store LFS objects in remote object storage which allows you to offload local hard disk R/W operations, and free up disk space significantly. @@ -91,6 +90,7 @@ Here is a configuration example with S3. | `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` | | `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` | | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | @@ -106,7 +106,9 @@ Here is a configuration example with GCS. | `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | | `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` | -_NOTE: The service account must have permission to access the bucket. [See more](https://cloud.google.com/storage/docs/authentication)_ +NOTE: **Note:** +The service account must have permission to access the bucket. +[See more](https://cloud.google.com/storage/docs/authentication) Here is a configuration example with Rackspace Cloud Files. @@ -118,7 +120,13 @@ Here is a configuration example with Rackspace Cloud Files. | `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | | `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | -_NOTES: Regardless of whether the container has public access enabled or disabled, Fog will use the TempURL method to grant access to LFS objects. If you see errors in logs referencing instantiating storage with a temp-url-key, ensure that you have set they key properly on the Rackspace API and in gitlab.rb. You can verify the value of the key Rackspace has set by sending a GET request with token header to the service access endpoint URL and comparing the output of the returned headers._ +NOTE: **Note:** +Regardless of whether the container has public access enabled or disabled, Fog will +use the TempURL method to grant access to LFS objects. If you see errors in logs referencing +instantiating storage with a temp-url-key, ensure that you have set they key properly +on the Rackspace API and in gitlab.rb. You can verify the value of the key Rackspace +has set by sending a GET request with token header to the service access endpoint URL +and comparing the output of the returned headers. ### Manual uploading to an object storage @@ -148,31 +156,31 @@ On Omnibus installations, the settings are prefixed by `lfs_object_store_`: 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: - ```ruby - gitlab_rails['lfs_object_store_enabled'] = true - gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" - gitlab_rails['lfs_object_store_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', - 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', - # The below options configure an S3 compatible host instead of AWS - 'host' => 'localhost', - 'endpoint' => 'http://127.0.0.1:9000', - 'path_style' => true - } - ``` + ```ruby + gitlab_rails['lfs_object_store_enabled'] = true + gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" + gitlab_rails['lfs_object_store_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N', + 'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE', + # The below options configure an S3 compatible host instead of AWS + 'host' => 'localhost', + 'endpoint' => 'http://127.0.0.1:9000', + 'path_style' => true + } + ``` 1. Save the file and [reconfigure GitLab]s for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: - ```bash - gitlab-rake gitlab:lfs:migrate - ``` + ```bash + gitlab-rake gitlab:lfs:migrate + ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless - `gitlab_rails['lfs_object_store_background_upload']` is set to false. + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless + `gitlab_rails['lfs_object_store_background_upload']` is set to false. ### S3 for installations from source @@ -182,33 +190,33 @@ For source installations the settings are nested under `lfs:` and then 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: - ```yaml - lfs: - enabled: true - object_store: - enabled: false - remote_directory: lfs-objects # Bucket name - connection: - provider: AWS - aws_access_key_id: 1ABCD2EFGHI34JKLM567N - aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE - region: eu-central-1 - # Use the following options to configure an AWS compatible host such as Minio - host: 'localhost' - endpoint: 'http://127.0.0.1:9000' - path_style: true - ``` + ```yaml + lfs: + enabled: true + object_store: + enabled: false + remote_directory: lfs-objects # Bucket name + connection: + provider: AWS + aws_access_key_id: 1ABCD2EFGHI34JKLM567N + aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE + region: eu-central-1 + # Use the following options to configure an AWS compatible host such as Minio + host: 'localhost' + endpoint: 'http://127.0.0.1:9000' + path_style: true + ``` 1. Save the file and [restart GitLab][] for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: - ```bash - sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production - ``` + ```bash + sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production + ``` - This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless `background_upload` is set to - false. + This will migrate existing LFS objects to object storage. New LFS objects + will be forwarded to object storage unless `background_upload` is set to + false. ## Storage statistics diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index 202f2e39975..f5593927cc2 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -35,9 +35,10 @@ Documentation for GitLab instance administrators is under [LFS administration do - Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have to add the URL to Git config manually (see [troubleshooting](#troubleshooting)) ->**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication - still goes over HTTP, but now the SSH client passes the correct credentials - to the Git LFS client, so no action is required by the user. +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. ## Using Git LFS @@ -61,12 +62,13 @@ git commit -am "Added Debian iso" # commit the file meta data git push origin master # sync the git repo and large file to the GitLab server ``` -> **Note**: Make sure that `.gitattributes` is tracked by git. Otherwise Git -> LFS will not be working properly for people cloning the project. -> -> ```bash -> git add .gitattributes -> ``` +NOTE: **Note:** +**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git +LFS will not be working properly for people cloning the project. + +```bash +git add .gitattributes +``` Cloning the repository works the same as before. Git automatically detects the LFS-tracked files and clones them via HTTP. If you performed the git clone @@ -84,6 +86,10 @@ that are on the remote repository, eg. for a branch from origin: git lfs fetch origin master ``` +### Migrate an existing repo to Git LFS + +Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md). + ## File Locking > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/35856) in GitLab 10.5. @@ -212,9 +218,10 @@ git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs" ### Credentials are always required when pushing an object ->**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication - still goes over HTTP, but now the SSH client passes the correct credentials - to the Git LFS client, so no action is required by the user. +NOTE: **Note:** +With 8.12 GitLab added LFS support to SSH. The Git LFS communication +still goes over HTTP, but now the SSH client passes the correct credentials +to the Git LFS client, so no action is required by the user. Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing the LFS object on every push for every object, user HTTPS credentials are required. @@ -250,7 +257,7 @@ If you are storing LFS files outside of GitLab you can disable LFS on the projec It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`. -You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store, +You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store, GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled. To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). diff --git a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md index 71c73e3dffe..5ceeb3253f1 100644 --- a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md +++ b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md @@ -46,25 +46,24 @@ need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab- repository and that you have made backups in case something goes wrong). Fire up a terminal, navigate to your Git repository and: - 1. Disable `git-annex`: - ```bash - git annex sync --content - git annex direct - git annex uninit - git annex indirect - ``` + ```bash + git annex sync --content + git annex direct + git annex uninit + git annex indirect + ``` 1. Enable `git-lfs`: - ``` - git lfs install - git lfs track <files> - git add . - git commit -m "commit message" - git push - ``` + ``` + git lfs install + git lfs track <files> + git add . + git commit -m "commit message" + git push + ``` ### Disabling Git Annex in your repo @@ -77,7 +76,7 @@ Here you'll find a guide on Since Annex files are stored as objects with symlinks and cannot be directly modified, we need to first remove those symlinks. ->**Note:** +NOTE: **Note:** Make sure the you read about the [`direct` mode][annex-direct] as it contains useful information that may fit in your use case. Note that `annex direct` is deprecated in Git Annex version 6, so you may need to upgrade your repository @@ -86,72 +85,72 @@ if the server also has Git Annex 6 installed. Read more in the 1. Backup your repository - ```bash - cd repository - git annex sync --content - cd .. - git clone repository repository-backup - cd repository-backup - git annex get - cd .. - ``` + ```bash + cd repository + git annex sync --content + cd .. + git clone repository repository-backup + cd repository-backup + git annex get + cd .. + ``` 1. Use `annex direct`: - ```bash - cd repository - git annex direct - ``` + ```bash + cd repository + git annex direct + ``` - The output should be similar to this: + The output should be similar to this: - ```bash - commit - On branch master - Your branch is up-to-date with 'origin/master'. - nothing to commit, working tree clean - ok - direct debian.iso ok - direct ok - ``` + ```bash + commit + On branch master + Your branch is up-to-date with 'origin/master'. + nothing to commit, working tree clean + ok + direct debian.iso ok + direct ok + ``` 1. Disable Git Annex with [`annex uninit`][uninit]: - ```bash - git annex uninit - ``` + ```bash + git annex uninit + ``` - The output should be similar to this: + The output should be similar to this: - ```bash - unannex debian.iso ok - Deleted branch git-annex (was 2534d2c). - ``` + ```bash + unannex debian.iso ok + Deleted branch git-annex (was 2534d2c). + ``` - This will `unannex` every file in the repository, leaving the original files. + This will `unannex` every file in the repository, leaving the original files. 1. Switch back to `indirect` mode: - ```bash - git annex indirect - ``` - - The output should be similar to this: + ```bash + git annex indirect + ``` - ```bash - (merging origin/git-annex into git-annex...) - (recording state in git...) - commit (recording state in git...) + The output should be similar to this: - ok - (recording state in git...) - [master fac3194] commit before switching to indirect mode - 1 file changed, 1 deletion(-) - delete mode 120000 alpine-virt-3.4.4-x86_64.iso - ok - indirect ok - ok - ``` + ```bash + (merging origin/git-annex into git-annex...) + (recording state in git...) + commit (recording state in git...) + + ok + (recording state in git...) + [master fac3194] commit before switching to indirect mode + 1 file changed, 1 deletion(-) + delete mode 120000 alpine-virt-3.4.4-x86_64.iso + ok + indirect ok + ok + ``` --- @@ -216,16 +215,16 @@ branches created by Git Annex: `git-annex`, and all under `synced/`.  -You can also do this on the commandline with: +You can also do this on the command line with: - ```bash - git branch -d synced/master - git branch -d synced/git-annex - git push origin :synced/master - git push origin :synced/git-annex - git push origin :git-annex - git remote prune origin - ``` +```bash +git branch -d synced/master +git branch -d synced/git-annex +git push origin :synced/master +git push origin :synced/git-annex +git push origin :git-annex +git remote prune origin +``` If there are still some Annex objects inside your repository (`.git/annex/`) or references inside `.git/config`, run `annex uninit` again: diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 1e8674f863d..5abae929355 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -51,6 +51,10 @@ Organization like this is suitable for users that belong to different groups but same need for being notified for every group they are member of. These settings can be configured on group page under the name of the group. It will be the dropdown with the bell icon. They can also be configured on the user profile notifications dropdown. +The group owner can disable email notifications for a group, which also includes +it's subgroups and projects. If this is the case, you will not receive any corresponding notifications, +and the notification button will be disabled with an explanatory tooltip. + ### Project Settings  @@ -60,6 +64,10 @@ other setting. This is suitable for users that have different needs for notifications per project basis. These settings can be configured on project page under the name of the project. It will be the dropdown with the bell icon. They can also be configured on the user profile notifications dropdown. +The project owner (or it's group owner) can disable email notifications for the project. +If this is the case, you will not receive any corresponding notifications, and the notification +button will be disabled with an explanatory tooltip. + ## Notification events Below is the table of events users can be notified of: @@ -138,7 +146,7 @@ Notification emails include headers that provide extra content about the notific | X-GitLab-Project-Id | The ID of the project | | X-GitLab-Project-Path | The path of the project | | X-GitLab-(Resource)-ID | The ID of the resource the notification is for, where resource is `Issue`, `MergeRequest`, `Commit`, etc| -| X-GitLab-Discussion-ID | Only in comment emails, the ID of the discussion the comment is from | +| X-GitLab-Discussion-ID | Only in comment emails, the ID of the thread the comment is from | | X-GitLab-Pipeline-Id | Only in pipeline emails, the ID of the pipeline the notification is for | | X-GitLab-Reply-Key | A unique token to support reply by email | | X-GitLab-NotificationReason | The reason for being notified. "mentioned", "assigned", etc | diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md index 87ca46e94be..753518d0424 100644 --- a/doc/workflow/repository_mirroring.md +++ b/doc/workflow/repository_mirroring.md @@ -37,8 +37,7 @@ The following are some possible use cases for repository mirroring: ## Pushing to a remote repository **(CORE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in GitLab Enterprise -> Edition 8.7. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/249) in GitLab Enterprise Edition 8.7. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. For an existing project, you can set up push mirroring as follows: @@ -67,8 +66,7 @@ section. ### Push only protected branches **(CORE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3350) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18715) in 10.8. You can choose to only push your protected branches from GitLab to your remote repository. @@ -92,14 +90,13 @@ The repository will push soon. To force a push, click the appropriate button. 1. On the destination GitLab instance, create a [personal access token](../user/profile/personal_access_tokens.md) with `API` scope. 1. On the source GitLab instance: - 1. Fill in the **Git repository URL** field using this format: `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. - 1. Fill in **Password** field with the GitLab personal access token created on the destination GitLab instance. - 1. Click the **Mirror repository** button. + 1. Fill in the **Git repository URL** field using this format: `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + 1. Fill in **Password** field with the GitLab personal access token created on the destination GitLab instance. + 1. Click the **Mirror repository** button. ## Pulling from a remote repository **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. -> [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab-ee/issues/10871) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab-ee/issues/10871) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.11. NOTE: **Note:** This feature [is available for free](https://gitlab.com/gitlab-org/gitlab-ee/issues/10361) to GitLab.com users until September 22nd, 2019. @@ -118,9 +115,9 @@ To configure mirror pulling for an existing project: 1. Select **Pull** from the **Mirror direction** dropdown. 1. Select an authentication method from the **Authentication method** dropdown, if necessary. 1. If necessary, check the following boxes: - - **Overwrite diverged branches**. - - **Trigger pipelines for mirror updates**. - - **Only mirror protected branches**. + - **Overwrite diverged branches**. + - **Trigger pipelines for mirror updates**. + - **Only mirror protected branches**. 1. Click the **Mirror repository** button to save the configuration.  @@ -157,8 +154,7 @@ Repository mirrors are updated as Sidekiq becomes available to process them. If ### SSH authentication -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551) for Pull mirroring in [GitLab Starter](https://about.gitlab.com/pricing/) 9.5. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22982) for Push mirroring in [GitLab Core](https://about.gitlab.com/pricing/) 11.6 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2551) for Pull mirroring in [GitLab Starter](https://about.gitlab.com/pricing/) 9.5. [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22982) for Push mirroring in [GitLab Core](https://about.gitlab.com/pricing/) 11.6 SSH authentication is mutual: @@ -245,8 +241,7 @@ key to keep the mirror running. ### Overwrite diverged branches **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4559) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.6. You can choose to always update your local branches with remote versions, even if they have diverged from the remote. @@ -258,8 +253,7 @@ To use this option, check the **Overwrite diverged branches** box when creating ### Only mirror protected branches **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3326) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. You can choose to pull mirror only the protected branches from your remote repository to GitLab. Non-protected branches are not mirrored and can diverge. @@ -268,8 +262,7 @@ To use this option, check the **Only mirror protected branches** box when creati ### Hard failure **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117) in -> [GitLab Starter](https://about.gitlab.com/pricing/) 10.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3117) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.2. Once the mirroring process is unsuccessfully retried 14 times in a row, it will get marked as hard failed. This will become visible in either the: @@ -282,8 +275,7 @@ project mirroring again by [Forcing an update](#forcing-an-update-core). ### Trigger update using API **(STARTER)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453) in -[GitLab Starter](https://about.gitlab.com/pricing/) 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3453) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. Pull mirroring uses polling to detect new branches and commits added upstream, often minutes afterwards. If you notify GitLab by [API](../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), @@ -325,9 +317,10 @@ protected branches. ### Preventing conflicts using a `pre-receive` hook -> **Warning:** The solution proposed will negatively impact the performance of -> Git push operations because they will be proxied to the upstream Git -> repository. +CAUTION: **Warning:** +The solution proposed will negatively impact the performance of +Git push operations because they will be proxied to the upstream Git +repository. A server-side `pre-receive` hook can be used to prevent the race condition described above by only accepting the push after first pushing the commit to diff --git a/doc/workflow/shortcuts.md b/doc/workflow/shortcuts.md index fd67ea8ce87..5d08bf5e77d 100644 --- a/doc/workflow/shortcuts.md +++ b/doc/workflow/shortcuts.md @@ -35,11 +35,11 @@ You can see GitLab's keyboard shortcuts by using <kbd>shift</kbd> + <kbd>?</kbd> | Keyboard Shortcut | Description | | ----------------- | ----------- | -| <kbd>g</kbd> + <kbd>a</kbd> | Go to the activity feed | -| <kbd>g</kbd> + <kbd>p</kbd> | Go to projects | -| <kbd>g</kbd> + <kbd>i</kbd> | Go to issues | -| <kbd>g</kbd> + <kbd>m</kbd> | Go to merge requests | -| <kbd>g</kbd> + <kbd>t</kbd> | Go to todos | +| <kbd>Shift</kbd> + <kbd>a</kbd> | Go to the activity feed | +| <kbd>Shift</kbd> + <kbd>p</kbd> | Go to projects | +| <kbd>Shift</kbd> + <kbd>i</kbd> | Go to issues | +| <kbd>Shift</kbd> + <kbd>m</kbd> | Go to merge requests | +| <kbd>Shift</kbd> + <kbd>t</kbd> | Go to todos | ## Project @@ -84,6 +84,8 @@ You can see GitLab's keyboard shortcuts by using <kbd>shift</kbd> + <kbd>?</kbd> | <kbd>l</kbd> | Change label | | <kbd>]</kbd> or <kbd>j</kbd> | Move to next file | | <kbd>[</kbd> or <kbd>k</kbd> | Move to previous file | +| <kbd>n</kbd> | Move to next unresolved discussion | +| <kbd>p</kbd> | Move to previous unresolved discussion | ## Epics **(ULTIMATE)** diff --git a/doc/workflow/time_tracking.md b/doc/workflow/time_tracking.md index 4286a3625a2..fad853f8a44 100644 --- a/doc/workflow/time_tracking.md +++ b/doc/workflow/time_tracking.md @@ -22,8 +22,8 @@ below. ## How to enter data -Time Tracking uses two [quick actions] that GitLab introduced with this new -feature: `/spend` and `/estimate`. +Time Tracking uses two [quick actions](../user/project/quick_actions.md) +that GitLab introduced with this new feature: `/spend` and `/estimate`. Quick actions can be used in the body of an issue or a merge request, but also in a comment in both an issue or a merge request. @@ -73,16 +73,15 @@ The following time units are available: Default conversion rates are 1mo = 4w, 1w = 5d and 1d = 8h. -### Limit displayed units to hours +### Limit displayed units to hours **(CORE ONLY)** -> Introduced in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29469/) in GitLab 12.1. -The display of time units can be limited to hours through the option in **Admin Area > Settings > Preferences** under 'Localization'. +In GitLab self-managed instances, the display of time units can be limited to +hours through the option in **Admin Area > Settings > Preferences** under **Localization**. With this option enabled, `75h` is displayed instead of `1w 4d 3h`. ## Other interesting links - [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/solutions/time-tracking/) - -[quick actions]: ../user/project/quick_actions.md diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 1f8900c734b..211d242e2d3 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -12,8 +12,6 @@ in a simple dashboard.  ---- - You can quickly access your To-Do List by clicking the checkmark icon next to the search bar in the top navigation. If the count is: @@ -31,15 +29,15 @@ A To Do displays on your To-Do List when: - An issue or merge request is assigned to you - You are `@mentioned` in the description or comment of an: - - Issue - - Merge Request - - Epic **(ULTIMATE)** + - Issue + - Merge Request + - Epic **(ULTIMATE)** - You are `@mentioned` in a comment on a commit - A job in the CI pipeline running for your merge request failed, but this job is not allowed to fail -- An open merge request becomes unmergeable due to conflict, and you are either: - - The author - - Have set it to automatically merge once the pipeline succeeds +- An open merge request becomes unmergeable due to conflict, and you are either: + - The author + - Have set it to automatically merge once the pipeline succeeds To-do triggers are not affected by [GitLab Notification Email settings](notifications.md). @@ -94,8 +92,6 @@ Actions that dismiss To-Do items include: - Adding/removing a label - Commenting on the issue ---- - Your To-Do List is personal, and items are only marked as done if the action comes from you. If you close the issue or merge request, your To Do is automatically marked as done. @@ -108,8 +104,6 @@ To prevent other users from closing issues without you being notified, if someon There is just one To Do for each of these, so mentioning a user a hundred times in an issue will only trigger one To Do. ---- - If no action is needed, you can manually mark the To Do as done by clicking the corresponding **Done** button, and it will disappear from your To-Do List. diff --git a/doc/workflow/workflow.md b/doc/workflow/workflow.md index f70e41df842..7fac41c3b6f 100644 --- a/doc/workflow/workflow.md +++ b/doc/workflow/workflow.md @@ -1,31 +1,31 @@ # Feature branch workflow -1. Clone project: +1. Clone project: - ```bash - git clone git@example.com:project-name.git - ``` + ```bash + git clone git@example.com:project-name.git + ``` -1. Create branch with your feature: +1. Create branch with your feature: - ```bash - git checkout -b $feature_name - ``` + ```bash + git checkout -b $feature_name + ``` -1. Write code. Commit changes: +1. Write code. Commit changes: - ```bash - git commit -am "My feature is ready" - ``` + ```bash + git commit -am "My feature is ready" + ``` -1. Push your branch to GitLab: +1. Push your branch to GitLab: - ```bash - git push origin $feature_name - ``` + ```bash + git push origin $feature_name + ``` -1. Review your code on commits page. +1. Review your code on commits page. -1. Create a merge request. +1. Create a merge request. -1. Your team lead will review the code & merge it to the main branch. +1. Your team lead will review the code & merge it to the main branch. |
