diff options
Diffstat (limited to 'doc/topics/autodevops')
31 files changed, 395 insertions, 147 deletions
diff --git a/doc/topics/autodevops/img/auto_monitoring.png b/doc/topics/autodevops/img/auto_monitoring.png Binary files differindex 92902e3ca72..2900e5d1877 100644 --- a/doc/topics/autodevops/img/auto_monitoring.png +++ b/doc/topics/autodevops/img/auto_monitoring.png diff --git a/doc/topics/autodevops/img/autodevops_domain_variables.png b/doc/topics/autodevops/img/autodevops_domain_variables.png Binary files differnew file mode 100644 index 00000000000..b6f8864796f --- /dev/null +++ b/doc/topics/autodevops/img/autodevops_domain_variables.png diff --git a/doc/topics/autodevops/img/autodevops_multiple_clusters.png b/doc/topics/autodevops/img/autodevops_multiple_clusters.png Binary files differnew file mode 100644 index 00000000000..f4d101ca921 --- /dev/null +++ b/doc/topics/autodevops/img/autodevops_multiple_clusters.png diff --git a/doc/topics/autodevops/img/guide_choose_gke.png b/doc/topics/autodevops/img/guide_choose_gke.png Binary files differnew file mode 100644 index 00000000000..6da3a7220da --- /dev/null +++ b/doc/topics/autodevops/img/guide_choose_gke.png diff --git a/doc/topics/autodevops/img/guide_cluster_apps.png b/doc/topics/autodevops/img/guide_cluster_apps.png Binary files differnew file mode 100644 index 00000000000..33d25f2950d --- /dev/null +++ b/doc/topics/autodevops/img/guide_cluster_apps.png diff --git a/doc/topics/autodevops/img/guide_connect_cluster.png b/doc/topics/autodevops/img/guide_connect_cluster.png Binary files differindex b856b81a1d0..703d536f37a 100644 --- a/doc/topics/autodevops/img/guide_connect_cluster.png +++ b/doc/topics/autodevops/img/guide_connect_cluster.png diff --git a/doc/topics/autodevops/img/guide_create_cluster.png b/doc/topics/autodevops/img/guide_create_cluster.png Binary files differnew file mode 100644 index 00000000000..cd1d0fdd8da --- /dev/null +++ b/doc/topics/autodevops/img/guide_create_cluster.png diff --git a/doc/topics/autodevops/img/guide_create_project.png b/doc/topics/autodevops/img/guide_create_project.png Binary files differnew file mode 100644 index 00000000000..4ed1071db03 --- /dev/null +++ b/doc/topics/autodevops/img/guide_create_project.png diff --git a/doc/topics/autodevops/img/guide_enable_autodevops.png b/doc/topics/autodevops/img/guide_enable_autodevops.png Binary files differnew file mode 100644 index 00000000000..0fc3ecca19a --- /dev/null +++ b/doc/topics/autodevops/img/guide_enable_autodevops.png diff --git a/doc/topics/autodevops/img/guide_environments.png b/doc/topics/autodevops/img/guide_environments.png Binary files differnew file mode 100644 index 00000000000..1d8d5614e64 --- /dev/null +++ b/doc/topics/autodevops/img/guide_environments.png diff --git a/doc/topics/autodevops/img/guide_environments_metrics.png b/doc/topics/autodevops/img/guide_environments_metrics.png Binary files differnew file mode 100644 index 00000000000..f0d31f31581 --- /dev/null +++ b/doc/topics/autodevops/img/guide_environments_metrics.png diff --git a/doc/topics/autodevops/img/guide_first_pipeline.png b/doc/topics/autodevops/img/guide_first_pipeline.png Binary files differnew file mode 100644 index 00000000000..57459dcc9d9 --- /dev/null +++ b/doc/topics/autodevops/img/guide_first_pipeline.png diff --git a/doc/topics/autodevops/img/guide_gitlab_gke_details.png b/doc/topics/autodevops/img/guide_gitlab_gke_details.png Binary files differnew file mode 100644 index 00000000000..bc5a53800f7 --- /dev/null +++ b/doc/topics/autodevops/img/guide_gitlab_gke_details.png diff --git a/doc/topics/autodevops/img/guide_gke_apis_after.png b/doc/topics/autodevops/img/guide_gke_apis_after.png Binary files differnew file mode 100644 index 00000000000..380de958867 --- /dev/null +++ b/doc/topics/autodevops/img/guide_gke_apis_after.png diff --git a/doc/topics/autodevops/img/guide_gke_apis_before.png b/doc/topics/autodevops/img/guide_gke_apis_before.png Binary files differnew file mode 100644 index 00000000000..d06fc707887 --- /dev/null +++ b/doc/topics/autodevops/img/guide_gke_apis_before.png diff --git a/doc/topics/autodevops/img/guide_google_auth.png b/doc/topics/autodevops/img/guide_google_auth.png Binary files differnew file mode 100644 index 00000000000..b97b2be9f15 --- /dev/null +++ b/doc/topics/autodevops/img/guide_google_auth.png diff --git a/doc/topics/autodevops/img/guide_google_signin.png b/doc/topics/autodevops/img/guide_google_signin.png Binary files differnew file mode 100644 index 00000000000..e59fc94bd4c --- /dev/null +++ b/doc/topics/autodevops/img/guide_google_signin.png diff --git a/doc/topics/autodevops/img/guide_ide_commit.png b/doc/topics/autodevops/img/guide_ide_commit.png Binary files differnew file mode 100644 index 00000000000..188f60f2a4b --- /dev/null +++ b/doc/topics/autodevops/img/guide_ide_commit.png diff --git a/doc/topics/autodevops/img/guide_integration.png b/doc/topics/autodevops/img/guide_integration.png Binary files differdeleted file mode 100644 index 723b2619ea2..00000000000 --- a/doc/topics/autodevops/img/guide_integration.png +++ /dev/null diff --git a/doc/topics/autodevops/img/guide_merge_request.png b/doc/topics/autodevops/img/guide_merge_request.png Binary files differnew file mode 100644 index 00000000000..d78e69be776 --- /dev/null +++ b/doc/topics/autodevops/img/guide_merge_request.png diff --git a/doc/topics/autodevops/img/guide_merge_request_ide.png b/doc/topics/autodevops/img/guide_merge_request_ide.png Binary files differnew file mode 100644 index 00000000000..c825b0849e1 --- /dev/null +++ b/doc/topics/autodevops/img/guide_merge_request_ide.png diff --git a/doc/topics/autodevops/img/guide_merge_request_review_app.png b/doc/topics/autodevops/img/guide_merge_request_review_app.png Binary files differnew file mode 100644 index 00000000000..1b9b854ddac --- /dev/null +++ b/doc/topics/autodevops/img/guide_merge_request_review_app.png diff --git a/doc/topics/autodevops/img/guide_pipeline_stages.png b/doc/topics/autodevops/img/guide_pipeline_stages.png Binary files differnew file mode 100644 index 00000000000..6e2f078152b --- /dev/null +++ b/doc/topics/autodevops/img/guide_pipeline_stages.png diff --git a/doc/topics/autodevops/img/guide_project_landing_page.png b/doc/topics/autodevops/img/guide_project_landing_page.png Binary files differnew file mode 100644 index 00000000000..4f8d2eb10b1 --- /dev/null +++ b/doc/topics/autodevops/img/guide_project_landing_page.png diff --git a/doc/topics/autodevops/img/guide_project_template.png b/doc/topics/autodevops/img/guide_project_template.png Binary files differnew file mode 100644 index 00000000000..298ac0f6fcf --- /dev/null +++ b/doc/topics/autodevops/img/guide_project_template.png diff --git a/doc/topics/autodevops/img/guide_secret.png b/doc/topics/autodevops/img/guide_secret.png Binary files differdeleted file mode 100644 index 01f5aa49908..00000000000 --- a/doc/topics/autodevops/img/guide_secret.png +++ /dev/null diff --git a/doc/topics/autodevops/img/rollout_staging_disabled.png b/doc/topics/autodevops/img/rollout_staging_disabled.png Binary files differindex 71e36b440f0..4c7c6768666 100644 --- a/doc/topics/autodevops/img/rollout_staging_disabled.png +++ b/doc/topics/autodevops/img/rollout_staging_disabled.png diff --git a/doc/topics/autodevops/img/rollout_staging_enabled.png b/doc/topics/autodevops/img/rollout_staging_enabled.png Binary files differindex d0d1d356627..f45c1c2cb37 100644 --- a/doc/topics/autodevops/img/rollout_staging_enabled.png +++ b/doc/topics/autodevops/img/rollout_staging_enabled.png diff --git a/doc/topics/autodevops/img/staging_enabled.png b/doc/topics/autodevops/img/staging_enabled.png Binary files differindex 0ef1a67d641..f0e0cd1cfcd 100644 --- a/doc/topics/autodevops/img/staging_enabled.png +++ b/doc/topics/autodevops/img/staging_enabled.png diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index efec365042a..bb323705ab3 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -1,6 +1,6 @@ # Auto DevOps -> [Introduced][ce-37115] in GitLab 10.0. +> [Introduced][ce-37115] in GitLab 10.0. Generally available on GitLab 11.0. Auto DevOps automatically detects, builds, tests, deploys, and monitors your applications. @@ -13,6 +13,12 @@ without needing to configure anything. Just push your code and GitLab takes care of everything else. This makes it easier to start new projects and brings consistency to how applications are set up throughout a company. +## Quick start + +If you are using GitLab.com, see the [quick start guide](quick_start_guide.md) +for using Auto DevOps with GitLab.com and a Kubernetes cluster on Google Kubernetes +Engine. + ## Comparison to application platforms and PaaS Auto DevOps provides functionality described by others as an application @@ -34,18 +40,19 @@ in a couple of ways: ## Features Comprised of a set of stages, Auto DevOps brings these best practices to your -project in an easy and automatic way: +project in a simple and automatic way: 1. [Auto Build](#auto-build) 1. [Auto Test](#auto-test) -1. [Auto Code Quality](#auto-code-quality) -1. [Auto SAST (Static Application Security Testing)](#auto-sast) -1. [Auto Dependency Scanning](#auto-dependency-scanning) +1. [Auto Code Quality](#auto-code-quality) **[STARTER]** +1. [Auto SAST (Static Application Security Testing)](#auto-sast) **[ULTIMATE]** +1. [Auto Dependency Scanning](#auto-dependency-scanning) **[ULTIMATE]** +1. [Auto License Management](#auto-license-management) **[ULTIMATE]** 1. [Auto Container Scanning](#auto-container-scanning) 1. [Auto Review Apps](#auto-review-apps) -1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast) +1. [Auto DAST (Dynamic Application Security Testing)](#auto-dast) **[ULTIMATE]** 1. [Auto Deploy](#auto-deploy) -1. [Auto Browser Performance Testing](#auto-browser-performance-testing) +1. [Auto Browser Performance Testing](#auto-browser-performance-testing) **[PREMIUM]** 1. [Auto Monitoring](#auto-monitoring) As Auto DevOps relies on many different components, it's good to have a basic @@ -62,7 +69,7 @@ Auto DevOps provides great defaults for all the stages; you can, however, For an overview on the creation of Auto DevOps, read the blog post [From 2/3 of the Self-Hosted Git Market, to the Next-Generation CI System, to Auto DevOps](https://about.gitlab.com/2017/06/29/whats-next-for-gitlab-ci/). -## Prerequisites +## Requirements TIP: **Tip:** For self-hosted installations, the easiest way to make use of Auto DevOps is to @@ -84,7 +91,7 @@ To make full use of Auto DevOps, you will need: for the entire GitLab instance, or [specific Runners](../../ci/runners/README.md#registering-a-specific-runner) that are assigned to specific projects. 1. **Base domain** (needed for Auto Review Apps and Auto Deploy) - You will need - a domain configured with wildcard DNS which is gonna be used by all of your + a domain configured with wildcard DNS which is going to be used by all of your Auto DevOps applications. [Read the specifics](#auto-devops-base-domain). 1. **Kubernetes** (needed for Auto Review Apps, Auto Deploy, and Auto Monitoring) - To enable deployments, you will need Kubernetes 1.5+. You need a [Kubernetes cluster][kubernetes-clusters] @@ -112,31 +119,31 @@ NOTE: **Note:** If you do not have Kubernetes or Prometheus installed, then Auto Review Apps, Auto Deploy, and Auto Monitoring will be silently skipped. -### Auto DevOps base domain +## Auto DevOps base domain The Auto DevOps base domain is required if you want to make use of [Auto -Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It is defined -either under the project's CI/CD settings while -[enabling Auto DevOps](#enabling-auto-devops) or in instance-wide settings in -the CI/CD section. -It can also be set at the project or group level as a variable, `AUTO_DEVOPS_DOMAIN`. +Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined +in three places: -A wildcard DNS A record matching the base domain is required, for example, +- either under the project's CI/CD settings while [enabling Auto DevOps](#enabling-auto-devops) +- or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section +- or at the project or group level as a variable: `AUTO_DEVOPS_DOMAIN` (required if you want to use [multiple clusters](#using-multiple-kubernetes-clusters)) + +A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: ``` *.example.com 3600 A 1.2.3.4 ``` -where `example.com` is the domain name under which the deployed apps will be served, +In this case, `example.com` is the domain name under which the deployed apps will be served, and `1.2.3.4` is the IP address of your load balancer; generally NGINX -([see prerequisites](#prerequisites)). How to set up the DNS record is beyond +([see requirements](#requirements)). How to set up the DNS record is beyond the scope of this document; you should check with your DNS provider. -Alternatively you can use free public services like [xip.io](http://xip.io) or -[nip.io](http://nip.io) which provide automatic wildcard DNS without any -configuration. Just set the Auto DevOps base domain to `1.2.3.4.xip.io` or -`1.2.3.4.nip.io`. +Alternatively you can use free public services like [nip.io](http://nip.io) +which provide automatic wildcard DNS without any configuration. Just set the +Auto DevOps base domain to `1.2.3.4.nip.io`. Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s). @@ -146,26 +153,70 @@ If GitLab is installed using the [GitLab Omnibus Helm Chart], there are two options: provide a static IP, or have one assigned. For more information see the relevant docs on the [network prerequisites](../../install/kubernetes/gitlab_omnibus.md#networking-prerequisites). -## Quick start +## Using multiple Kubernetes clusters **[PREMIUM]** + +When using Auto DevOps, you may want to deploy different environments to +different Kubernetes clusters. This is possible due to the 1:1 connection that +[exists between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters). + +In the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml) +(used behind the scenes by Auto DevOps), there are currently 3 defined environment names that you need to know: + +- `review/` (every environment starting with `review/`) +- `staging` +- `production` + +Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so +except for the environment scope, they would also need to have a different +domain they would be deployed to. This is why you need to define a separate +`AUTO_DEVOPS_DOMAIN` variable for all the above +[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-variables). -If you are using GitLab.com, see our [quick start guide](quick_start_guide.md) -for using Auto DevOps with GitLab.com and an external Kubernetes cluster on -Google Cloud. +The following table is an example of how the three different clusters would +be configured. + +| Cluster name | Cluster environment scope | `AUTO_DEVOPS_DOMAIN` variable value | Variable environment scope | Notes | +| ------------ | -------------- | ----------------------------- | ------------- | ------ | +| review | `review/*` | `review.example.com` | `review/*` | The review cluster which will run all [Review Apps](../../ci/review_apps/index.md). `*` is a wildcard, which means it will be used by every environment name starting with `review/`. | +| staging | `staging` | `staging.example.com` | `staging` | (Optional) The staging cluster which will run the deployments of the staging environments. You need to [enable it first](#deploy-policy-for-staging-and-production-environments). | +| production | `production` | `example.com` | `production` | The production cluster which will run the deployments of the production environment. You can use [incremental rollouts](#incremental-rollout-to-production). | + +To add a different cluster for each environment: + +1. Navigate to your project's **Operations > Kubernetes** and create the Kubernetes clusters + with their respective environment scope as described from the table above. + +  + +1. After the clusters are created, navigate to each one and install Helm Tiller + and Ingress. +1. Make sure you have [configured your DNS](#auto-devops-base-domain) with the + specified Auto DevOps domains. +1. Navigate to your project's **Settings > CI/CD > Variables** and add + the `AUTO_DEVOPS_DOMAIN` variables with their respective environment + scope. + +  + +Now that all is configured, you can test your setup by creating a merge request +and verifying that your app is deployed as a review app in the Kubernetes +cluster with the `review/*` environment scope. Similarly, you can check the +other environments. ## Enabling Auto DevOps -If you haven't done already, read the [prerequisites](#prerequisites) to make +If you haven't done already, read the [requirements](#requirements) to make full use of Auto DevOps. If this is your fist time, we recommend you follow the -[quick start guide](#quick-start). +[quick start guide](quick_start_guide.md). To enable Auto DevOps to your project: -1. Check that your project doesn't have a `.gitlab-ci.yml`, and remove it otherwise -1. Go to your project's **Settings > CI/CD > General pipelines settings** and - find the Auto DevOps section +1. Check that your project doesn't have a `.gitlab-ci.yml`, or remove it otherwise +1. Go to your project's **Settings > CI/CD > Auto DevOps** 1. Select "Enable Auto DevOps" 1. Optionally, but recommended, add in the [base domain](#auto-devops-base-domain) - that will be used by Kubernetes to deploy your application + that will be used by Kubernetes to [deploy your application](#auto-deploy) + and choose the [deployment strategy](#deployment-strategy) 1. Hit **Save changes** for the changes to take effect Once saved, an Auto DevOps pipeline will be triggered on the default branch. @@ -182,6 +233,24 @@ in **Admin Area > Settings > Continuous Integration and Deployment**. Doing that all the projects that haven't explicitly set an option will have Auto DevOps enabled by default. +### Deployment strategy + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/38542) in GitLab 11.0. + +You can change the deployment strategy used by Auto DevOps by going to your +project's **Settings > CI/CD > Auto DevOps**. + +The available options are: + +- **Continuous deployment to production** - enables [Auto Deploy](#auto-deploy) + by setting the [`STAGING_ENABLED`](#deploy-policy-for-staging-and-production-environments) and + [`INCREMENTAL_ROLLOUT_ENABLED`](#incremental-rollout-to-production) variables + to false. +- **Automatic deployment to staging, manual deployment to production** - sets the + [`STAGING_ENABLED`](#deploy-policy-for-staging-and-production-environments) and + [`INCREMENTAL_ROLLOUT_ENABLED`](#incremental-rollout-to-production) variables + to true, and the user is responsible for manually deploying to staging and production. + ## Stages of Auto DevOps The following sections describe the stages of Auto DevOps. Read them carefully @@ -218,7 +287,7 @@ NOTE: **Note:** Auto Test uses tests you already have in your application. If there are no tests, it's up to you to add them. -### Auto Code Quality +### Auto Code Quality **[STARTER]** Auto Code Quality uses the [Code Quality image](https://gitlab.com/gitlab-org/security-products/codequality) to run @@ -230,7 +299,7 @@ In GitLab Starter, differences between the source and target branches are also [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html). -### Auto SAST +### Auto SAST **[ULTIMATE]** > Introduced in [GitLab Ultimate][ee] 10.3. @@ -241,9 +310,9 @@ report is created, it's uploaded as an artifact which you can later download and check out. In GitLab Ultimate, any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html). +[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/sast.html). -### Auto Dependency Scanning +### Auto Dependency Scanning **[ULTIMATE]** > Introduced in [GitLab Ultimate][ee] 10.7. @@ -253,8 +322,21 @@ to run analysis on the project dependencies and checks for potential security is report is created, it's uploaded as an artifact which you can later download and check out. -In GitLab Ultimate, any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html). +Any security warnings are also +[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dependency_scanning.html). + +### Auto License Management **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 11.0. + +License Management uses the +[License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +to search the project dependencies for their license. Once the +report is created, it's uploaded as an artifact which you can later download and +check out. + +Any licenses are also +[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/license_management.html). ### Auto Container Scanning @@ -267,13 +349,13 @@ created, it's uploaded as an artifact which you can later download and check out. In GitLab Ultimate, any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). +[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/container_scanning.html). ### Auto Review Apps NOTE: **Note:** This is an optional step, since many projects do not have a Kubernetes cluster -available. If the [prerequisites](#prerequisites) are not met, the job will +available. If the [requirements](#requirements) are not met, the job will silently be skipped. CAUTION: **Caution:** @@ -295,7 +377,7 @@ up in the merge request widget for easy discovery. When the branch is deleted, for example after the merge request is merged, the Review App will automatically be deleted. -### Auto DAST +### Auto DAST **[ULTIMATE]** > Introduced in [GitLab Ultimate][ee] 10.4. @@ -306,9 +388,9 @@ issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. In GitLab Ultimate, any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). +[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dast.html). -### Auto Browser Performance Testing +### Auto Browser Performance Testing **[PREMIUM]** > Introduced in [GitLab Premium][ee] 10.4. @@ -320,13 +402,14 @@ Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://h /direction ``` -In GitLab Premium, performance differences between the source and target branches are [shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). +In GitLab Premium, performance differences between the source +and target branches are [shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html). ### Auto Deploy NOTE: **Note:** This is an optional step, since many projects do not have a Kubernetes cluster -available. If the [prerequisites](#prerequisites) are not met, the job will +available. If the [requirements](#requirements) are not met, the job will silently be skipped. CAUTION: **Caution:** @@ -360,10 +443,19 @@ no longer be valid as soon as the deployment job finishes. This means that Kubernetes can run the application, but in case it should be restarted or executed somewhere else, it cannot be accessed again. +> [Introduced][ce-19507] in GitLab 11.0. + +For internal and private projects a [GitLab Deploy Token](../../user/project/deploy_tokens/index.md###gitlab-deploy-token) +will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved. This Deploy Token +can be used for permanent access to the registry. + +Note: **Note** +When the GitLab Deploy Token has been manually revoked, it won't be automatically created. + ### Auto Monitoring NOTE: **Note:** -Check the [prerequisites](#prerequisites) for Auto Monitoring to make this stage +Check the [requirements](#requirements) for Auto Monitoring to make this stage work. Once your application is deployed, Auto Monitoring makes it possible to monitor @@ -493,6 +585,8 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `POSTGRES_PASSWORD` | The PostgreSQL password; defaults to `testing-password`. Set it to use a custom password. | | `POSTGRES_DB` | The PostgreSQL database name; defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-variables-environment-variables). Set it to use a custom database name. | | `BUILDPACK_URL` | The buildpack's full URL. It can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`, for example `https://github.com/heroku/heroku-buildpack-ruby.git#v142` | +| `SAST_CONFIDENCE_LEVEL` | The minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High; defaults to `3`.| +| `DEP_SCAN_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled; defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).| | `STAGING_ENABLED` | From GitLab 10.8, this variable can be used to define a [deploy policy for staging and production environments](#deploy-policy-for-staging-and-production-environments). | | `CANARY_ENABLED` | From GitLab 11.0, this variable can be used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). | | `INCREMENTAL_ROLLOUT_ENABLED`| From GitLab 10.8, this variable can be used to enable an [incremental rollout](#incremental-rollout-to-production) of your application for the production environment. | @@ -575,6 +669,9 @@ service: > [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/160) in GitLab 10.8. +TIP: **Tip:** +You can also set this inside your [project's settings](#deployment-strategy). + The normal behavior of Auto DevOps is to use Continuous Deployment, pushing automatically to the `production` environment every time a new pipeline is run on the default branch. However, there are cases where you might want to use a @@ -605,6 +702,9 @@ If `CANARY_ENABLED` is defined in your project (e.g., set `CANARY_ENABLED` to > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5415) in GitLab 10.8. +TIP: **Tip:** +You can also set this inside your [project's settings](#deployment-strategy). + When you have a new version of your app to deploy in production, you may want to use an incremental rollout to replace just a few pods with the latest code. This will allow you to first check how the app is behaving, and later manually @@ -740,4 +840,5 @@ curl --data "value=true" --header "PRIVATE-TOKEN: personal_access_token" https:/ [postgresql]: https://www.postgresql.org/ [Auto DevOps template]: https://gitlab.com/gitlab-org/gitlab-ci-yml/blob/master/Auto-DevOps.gitlab-ci.yml [GitLab Omnibus Helm Chart]: ../../install/kubernetes/gitlab_omnibus.md -[ee]: https://about.gitlab.com/products/ +[ee]: https://about.gitlab.com/pricing/ +[ce-19507]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19507 diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 61c04f3d9bb..44b0cf758dc 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -1,143 +1,290 @@ -# Auto DevOps: quick start guide +# Getting started with Auto DevOps -> [Introduced][ce-37115] in GitLab 10.0. +This is a step-by-step guide that will help you use [Auto DevOps](index.md) to +deploy a project hosted on GitLab.com to Google Kubernetes Engine. -This is a step-by-step guide to deploying a project hosted on GitLab.com to -Google Cloud, using Auto DevOps. +We will use GitLab's native Kubernetes integration, so you will not need +to create a Kubernetes cluster manually using the Google Cloud Platform console. +We will create and deploy a simple application that we create from a GitLab template. -We made a minimal [Ruby -application](https://gitlab.com/auto-devops-examples/minimal-ruby-app) to use -as an example for this guide. It contains two main files: +These instructions will also work for a self-hosted GitLab instance; you'll just +need to ensure your own [Runners are configured](../../ci/runners/README.md) and +[Google OAuth is enabled](../../integration/google.md). -* `server.rb` - our application. It will start an HTTP server on port 5000 and - render "Hello, world!" -* `Dockerfile` - to build our app into a container image. It will use a ruby - base image and run `server.rb` +## Configuring your Google account -## Fork sample project on GitLab.com +Before creating and connecting your Kubernetes cluster to your GitLab project, +you need a Google Cloud Platform account. If you don't already have one, +sign up at https://console.cloud.google.com. You'll need to either sign in with an existing +Google account (for example, one that you use to access Gmail, Drive, etc.) or create a new one. -Let’s start by forking our sample application. Go to [the project -page](https://gitlab.com/auto-devops-examples/minimal-ruby-app) and press the -**Fork** button. Soon you should have a project under your namespace with the -necessary files. +1. Follow the steps as outlined in the ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin) + in order for the required APIs and related services to be enabled. +1. Make sure you have created a [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account). -You can also start a new project from a -[GitLab project template](https://gitlab.com/gitlab-org/project-templates) if -you want to use a different language. +TIP: **Tip:** +Every new Google Cloud Platform (GCP) account receives [$300 in credit](https://console.cloud.google.com/freetrial), +and in partnership with Google, GitLab is able to offer an additional $200 for new GCP accounts to get started with GitLab's +Google Kubernetes Engine Integration. All you have to do is [follow this link](https://goo.gl/AaJzRW) and apply for credit. -## Setup your own cluster on Google Kubernetes Engine +## Creating a new project from a template -If you do not already have a Google Cloud account, create one at -https://console.cloud.google.com. +We will use one of GitLab's project templates to get started. As the name suggests, +those projects provide a barebones application built on some well-known frameworks. -Visit the [**Kubernetes Engine**](https://console.cloud.google.com/kubernetes/list) -tab and create a new cluster. You can change the name and leave the rest of the -default settings. Once you have your cluster running, you need to connect to the -cluster by following the Google interface. +1. In GitLab, click the plus icon (**+**) at the top of the navigation bar and select + **New project**. +1. Go to the **Create from template** tab where you can choose among a Ruby on + Rails, Spring, or NodeJS Express project. For this example, + we'll use the Ruby on Rails template. -## Connect to Kubernetes cluster +  -You need to have the Google Cloud SDK installed. e.g. -On macOS, install [homebrew](https://brew.sh): +1. Give your project a name, optionally a description, and make it public so that + you can take advantage of the features available in the + [GitLab Gold plan](https://about.gitlab.com/pricing/#gitlab-com). -1. Install Brew Caskroom: `brew install caskroom/cask/brew-cask` -2. Install Google Cloud SDK: `brew cask install google-cloud-sdk` -3. Add `kubectl` with: `gcloud components install kubectl` -4. Log in: `gcloud auth login` +  -Now go back to the Google interface, find your cluster, follow the instructions -under "Connect to the cluster" and open the Kubernetes Dashboard. It will look -something like: +1. Click **Create project**. -```sh -gcloud container clusters get-credentials ruby-autodeploy \ --zone europe-west2-c --project api-project-XXXXXXX -``` +Now that the project is created, the next step is to create the Kubernetes cluster +under which this application will be deployed. -Finally, run `kubectl proxy`. +## Creating a Kubernetes cluster from within GitLab - +1. On the project's landing page, click the button labeled **Add Kubernetes cluster** + (note that this option is also available when you navigate to **Operations > Kubernetes**). -## Copy credentials to GitLab.com project +  -Once you have the Kubernetes Dashboard interface running, you should visit -**Secrets** under the "Config" section. There, you should find the settings we -need for GitLab integration: `ca.crt` and token. +1. Choose **Create on Google Kubernetes Engine**. - +  -You need to copy-paste the `ca.crt` and token into your project on GitLab.com in -the Kubernetes integration page under project -**Settings > Integrations > Project services > Kubernetes**. Don't actually copy -the namespace though. Each project should have a unique namespace, and by leaving -it blank, GitLab will create one for you. +1. Sign in with Google. - +  -For the API URL, you should use the "Endpoint" IP from your cluster page on -Google Cloud Platform. +1. Connect with your Google account and press **Allow** when asked (this will + be shown only the first time you connect GitLab with your Google account). -## Expose application to the world +  -In order to be able to visit your application, you need to install an NGINX -ingress controller and point your domain name to its external IP address. Let's -see how that's done. +1. The last step is to fill in the cluster details. Give it a name, leave the + environment scope as is, and choose the GCP project under which the cluster + will be created. (Per the instructions when you + [configured your Google account](#configuring-your-google-account), a project + should have already been created for you.) Next, choose the + [region/zone](https://cloud.google.com/compute/docs/regions-zones/) under which the + cluster will be created, enter the number of nodes you want it to have, and + finally choose their [machine type](https://cloud.google.com/compute/docs/machine-types). -### Set up Ingress controller +  -You’ll need to make sure you have an ingress controller. If you don’t have one, do: +1. Once ready, click **Create Kubernetes cluster**. -```sh -brew install kubernetes-helm -helm init -helm install --name ruby-app stable/nginx-ingress -``` +After a couple of minutes, the cluster will be created. You can also see its +status on your [GCP dashboard](https://console.cloud.google.com/kubernetes). -This should create several services including `ruby-app-nginx-ingress-controller`. -You can list your services by running `kubectl get svc` to confirm that. +The next step is to install some applications on your cluster that are needed +to take full advantage of Auto DevOps. -### Point DNS at Cluster IP +## Installing Helm, Ingress, and Prometheus -Find out the external IP address of the `ruby-app-nginx-ingress-controller` by -running: +GitLab's Kubernetes integration comes with some +[pre-defined applications](../../user/project/clusters/index.md#installing-applications) +for you to install. -```sh -kubectl get svc ruby-app-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' -``` + + +The first one to install is Helm Tiller, a package manager for Kubernetes, which +is needed in order to install the rest of the applications. Go ahead and click +its **Install** button. + +Once it's installed, the other applications that rely on it will each have their **Install** +button enabled. For this guide, we need Ingress and Prometheus. Ingress provides +load balancing, SSL termination, and name-based virtual hosting, using NGINX behind +the scenes. Prometheus is an open-source monitoring and alerting system that we'll +use to supervise the deployed application. We will not install GitLab Runner as +we'll use the shared Runners that GitLab.com provides. + +After the Ingress is installed, wait a few seconds and copy the IP address that +is displayed, which we'll use in the next step when enabling Auto DevOps. + +## Enabling Auto DevOps + +Now that the Kubernetes cluster is set up and ready, let's enable Auto DevOps. + +1. First, navigate to **Settings > CI/CD > Auto DevOps**. +1. Select **Enable Auto DevOps**. +1. Add in your base **Domain** by using the one GitLab suggests. Note that + generally, you would associate the IP address with a domain name on your + registrar's settings. In this case, for the sake of the guide, we will use + an alternative DNS that will map any domain name of the scheme + `anything.ip_address.nip.io` to the corresponding `ip_address`. For example, + if the IP address of the Ingress is `1.2.3.4`, the domain name to fill in + would be `1.2.3.4.nip.io`. +1. Lastly, let's select the [continuous deployment strategy](index.md#deployment-strategy) + which will automatically deploy the application to production once the pipeline + successfully runs on the `master` branch. +1. Click **Save changes**. + +  + +Once you complete all the above and save your changes, a new pipeline is +automatically created. To view the pipeline, go to **CI/CD > Pipelines**. + + + +In the next section we'll break down the pipeline and explain what each job does. + +## Deploying the application + +By now you should see the pipeline running, but what is it running exactly? + +To navigate inside the pipeline, click its status badge. (It's status should be "running"). +The pipeline is split into 4 stages, each running a couple of jobs. + + + +In the **build** stage, the application is built into a Docker image and then +uploaded to your project's [Container Registry](../../user/project/container_registry.md) ([Auto Build](index.md#auto-build)). + +In the **test** stage, GitLab runs various checks on the application: + +- The `test` job runs unit and integration tests by detecting the language and + framework ([Auto Test](index.md#auto-test)) +- The `code_quality` job checks the code quality and is allowed to fail + ([Auto Code Quality](index.md#auto-code-quality)) **[STARTER]** +- The `container_scanning` job checks the Docker container if it has any + vulnerabilities and is allowed to fail ([Auto Container Scanning](index.md#auto-container-scanning)) +- The `dependency_scanning` job checks if the application has any dependencies + susceptible to vulnerabilities and is allowed to fail ([Auto Dependency Scanning](index.md#auto-dependency-scanning)) **[ULTIMATE]** +- The `sast` job runs static analysis on the current code to check for potential + security issues and is allowed to fail([Auto SAST](index.md#auto-sast)) **[ULTIMATE]** +- The `license_management` job searches the application's dependencies to determine each of their + licenses and is allowed to fail ([Auto License Management](index.md#auto-license-management)) **[ULTIMATE]** NOTE: **Note:** -If your ingress controller has been installed in a different way, you can find -how to get the external IP address in the -[Cluster documentation](../../user/project/clusters/index.md#getting-the-external-ip-address). +As you might have noticed, all jobs except `test` are allowed to fail in the +test stage. + +The **production** stage is run after the tests and checks finish, and it automatically +deploys the application in Kubernetes ([Auto Deploy](index.md#auto-deploy)). + +Lastly, in the **performance** stage, some performance tests will run +on the deployed application +([Auto Browser Performance Testing](index.md#auto-browser-performance-testing)). **[PREMIUM]** + +--- -Use this IP address to configure your DNS. This part heavily depends on your -preferences and domain provider. But in case you are not sure, just create an -A record with a wildcard host like `*.<your-domain>`. +The URL for the deployed application can be found under the **Environments** +page where you can also monitor your application. Let's explore that. + +### Monitoring + +Now that the application is successfully deployed, let's navigate to its +website. First, go to **Operations > Environments**. + + + +In **Environments** you can see some details about the deployed +applications. In the rightmost column for the production environment, you can make use of the three icons: + +- The first icon will open the URL of the application that is deployed in + production. It's a very simple page, but the important part is that it works! +- The next icon with the small graph will take you to the metrics page where + Prometheus collects data about the Kubernetes cluster and how the application + affects it (in terms of memory/CPU usage, latency, etc.). + +  + +- The third icon is the [web terminal](../../ci/environments.md#web-terminals) + and it will open a terminal session right inside the container where the + application is running. + +Right below, there is the +[Deploy Board](https://docs.gitlab.com/ee/user/project/deploy_boards.md). +The squares represent pods in your Kubernetes cluster that are associated with +the given environment. Hovering above each square you can see the state of a +deployment and clicking a square will take you to the pod's logs page. + +TIP: **Tip:** +There is only one pod hosting the application at the moment, but you can add +more pods by defining the [`REPLICAS` variable](index.md#environment-variables) +under **Settings > CI/CD > Variables**. + +### Working with branches + +Following the [GitLab flow](../../workflow/gitlab_flow.md#working-with-feature-branches) +let's create a feature branch that will add some content to the application. + +Under your repository, navigate to the following file: `app/views/welcome/index.html.erb`. +By now, it should only contain a paragraph: `<p>You're on Rails!</p>`, so let's +start adding content. Let's use GitLab's [Web IDE](../../user/project/web_ide/index.md) to make the change. Once +you're on the Web IDE, make the following change: + +```html +<p>You're on Rails! Powered by GitLab Auto DevOps.</p> +``` + +Stage the file, add a commit message, and create a new branch and a merge request +by clicking **Commit**. + + + +Once you submit the merge request, you'll see the pipeline running. This will +run all the jobs as [described previously](#deploying-the-application), as well +a few more that run only on branches other than `master`. + + + +After a few minutes you'll notice that there was a failure in a test. +This means there's a test that was 'broken' by our change. +Navigating into the `test` job that failed, you can see what the broken test is: + +``` +Failure: +WelcomeControllerTest#test_should_get_index [/app/test/controllers/welcome_controller_test.rb:7]: +<You're on Rails!> expected but was +<You're on Rails! Powered by GitLab Auto DevOps.>.. +Expected 0 to be >= 1. + +bin/rails test test/controllers/welcome_controller_test.rb:4 +``` -Use `nslookup minimal-ruby-app-staging.<yourdomain>` to confirm that domain is -assigned to the cluster IP. +Let's fix that: -## Set up Auto DevOps +1. Back to the merge request, click the **Web IDE** button. +1. Find the `test/controllers/welcome_controller_test.rb` file and open it. +1. Change line 7 to say `You're on Rails! Powered by GitLab Auto DevOps.` +1. Click **Commit**. +1. On your left, under "Unstaged changes", click the little checkmark icon + to stage the changes. +1. Write a commit message and click **Commit**. -In your GitLab.com project, go to **Settings > CI/CD** and find the Auto DevOps -section. Select "Enable Auto DevOps", add in your base domain, and save. +Now, if you go back to the merge request you should not only see the test passing, +but also the application deployed as a [review app](index.md#auto-review-apps). You +can visit it by following the URL in the merge request. The changes that we +previously made should be there. -Next, a pipeline needs to be triggered. Since the test project doesn't have a -`.gitlab-ci.yml`, you need to either push a change to the repository or -manually visit `https://gitlab.com/<username>/minimal-ruby-app/pipelines/new`, -where `<username>` is your username. + -This will create a new pipeline with several jobs: `build`, `test`, `code_quality`, -and `production`. The `build` job will create a Docker image with your new -change and push it to the Container Registry. The `test` job will test your -changes, whereas the `code_quality` job will run static analysis on your changes. -Finally, the `production` job will deploy your changes to a production application. +Once you merge the merge request, the pipeline will run on the `master` branch, +and the application will be eventually deployed straight to production. -Once the deploy job succeeds you should be able to see your application by -visiting the Kubernetes dashboard. Select the namespace of your project, which -will look like `minimal-ruby-app-23`, but with a unique ID for your project, -and your app will be listed as "production" under the Deployment tab. +## Conclusion -Once its ready, just visit `http://minimal-ruby-app.example.com` to see the -famous "Hello, world!"! +After implementing this project, you should now have a solid understanding of the basics of Auto DevOps. +We started from building and testing to deploying and monitoring an application +all within GitLab. Despite its automatic nature, Audo DevOps can also be configured +and customized to fit your workflow. Here are some helpful resources for further reading: -[ce-37115]: https://gitlab.com/gitlab-org/gitlab-ce/issues/37115 +1. [Auto DevOps](index.md) +1. [Multiple Kubernetes clusters](index.md#using-multiple-kubernetes-clusters) **[PREMIUM]** +1. [Incremental rollout to production](index.md#incremental-rollout-to-production) **[PREMIUM]** +1. [Disable jobs you don't need with environment variables](index.md#environment-variables) +1. [Use a static IP for your cluster](../../user/project/clusters/index.md#using-a-static-ip) +1. [Use your own buildpacks to build your application](index.md#custom-buildpacks) +1. [Prometheus monitoring](../../user/project/integrations/prometheus.md) |
